Arable API 2.3.0

Release Notes

Release Overview

Release Date: August 12, 2020

Instance Affected: Production

3rd Party Sensors: Sentek Drill and Drop Soil Moisture Probe DD-MTS and DD-MT, Davis Wind Anemometer 6410 and 7911

Type of Release: Minor

Version: 2.3.0

Release Overview

Arable API 2.3.0 is a minor release with updates to the NDVI spectral kernel calibrations and the Chlorophyll Index calculation, which will improve Arable Mark 2 measurement quality. Additions to the API at https://developer.arable.com/ include a new precip hours count and a data quality flag.

As in the previous release, some of the new features (Alert subscription routes, GDD growing season routes, and embedded BI analytics dashboard routes) are currently only available to the Arable frontend while we let them mature before publishing at https://developer.arable.com/.

New Features

A data quality flag has been added to the hourly and daily data endpoints. There are two new columns, low_quality and sample_pct, to help identify data with quality issues. In this release two validations are done:
1. Individual sensors that report out of range of what is physically possible and get the low quality flag raised.
2. Hourly aggregations that do not have sufficient number of data points, e.g., the number of individual hourly readings that went into the daily aggregation is less than 2/3 and marked as a low-quality measurement. There can be multiple errors for a single aggregation.
More quality criteria will be added in upcoming releases including time consistency, as will the option to fetch the explanation of a low-quality score.
A new parameter precip_hours has been added to the daily endpoint. This parameter captures the number of hours with more than 0.1 mm rain per day. Measurement unit is mm.
The chlorophyll index equation has changed in this release to use the green band formulation. See Gitelson and Merzlyak, 2005 for more details. Upon initial deployment you may see a jump in CI value as the data range changes from around 0-1 to about 0-20 (not a strict limit), with the Green-CI equation. Processing older data with a new equation will occur shortly after deployment. The end result will be a more robust CI measurement.
Spectral kernel calibration updates for Arable Mark 2 are also in this release, based on a thorough frequency response characterization using tunable light sources. As a result, you may see a smaller change in values in exports, and in the derived NDVI measurement and graph. As for older data for CI, we will be reprocessing spectral and NDVI historical values shortly after deployment.
Historical remote-sensed wind speed and wind direction from IBM/The Weather Company is now present in the API under locations/{location_id}/historical. The wind direction cardinality denotes where the wind comes from (e.g., Westerly winds). There may be up to 6 hours’ delay due to processing and cleaning of data by the remote sense data provider (IBM/The Weather Company). The data is finally validated after 24 hours has passed. The remote sensed data is given at 10-meter height in this release as provided by the source. We are working on validating the best process to downscale to the height of the Arable Mark for a future upgrade.

Defect Fixes

In some cases, we had seen data gaps for specific devices based on humidity and temp sensor board response delays associated with ‘reset8s.’ Firmware v1.1.0.10 has now improved this area and the backend was updated to handle seamless data transition across the midnight hour.
We have seen an individual Mark 2 that had its daily precipitation in the summary endpoint not providing the same value as when aggregating the data in the hourly data endpoint. This is now corrected.
The backend showed precipitation hourly data as null if no audio was discovered by the Mark 2. This created confusion as it was interpreted as data loss. We have now updated the API endpoints so that this case is showing up as “0” precipitation.
In the previous release, the summary endpoint for precipitation could run into a caching issue and not provide the latest updates as the daily endpoint does. This is now aligned and shows the same data updates on both.
There were individual Mark 2 units that were equipped with Bridge and Sentek soil moisture probes that got the flag set to Bridge=False after the last Arable Backend 2.2.0 upgrade. This has now been worked through so it can be avoided in new releases.

Known Issues

Wind data provided by the local Davis anemometer can be skewed in some cases due to wrong mapping from pulse count to degrees. This is being worked on and is expected to be updated with a hotfix once we have worked through the problem.
Remote-sensed wind data takes some time to process before being presented, which can lead to up to 6 hours of wind data not being available at the time of reading. This wind data will fill in again as time elapses. Future enhancements will provide current, remote-sensed, on-demand data.
Remote-sensed wind in API 2.3.0 is at 10 meters height. In a coming release, this will be lowered to 2 meters to be more aligned with the observed data from any Davis anemometer connected to the Arable Mark.
The current implementation to get the first hour of the weather forecast from IBM/The Weather Company is based on retrieving the remaining 15-minute forecast within the hour in real-time, as the current hour is not provided by the API on-demand. This means that we will not always have a full hour of data for temperature and precipitation based on API call timing. We plan to completely resolve this issue by caching the first hour forecast for all Arable Marks in an active state. This is planned for a future release. The current timing is as follows:
00-07 min after the hour, we will get a full hour of data.
08-22 min after the hour, we get 45 min of the hour’s data.
23-37 min after the hour, we get 30 min of the hour’s data
38-52 min after the hour, we get 15 min of the hour’s data
53-59 min after the hour, we get 0 min of the hour’s data
It can take a long time to see measurement data after a new installation if the unit was not properly undeployed after the previous installation, as data will have been collected during transport and the Mark will send its data in FIFO mode.
There is currently no way for customers to export all locations of a device ID, nor to delete a location via the API if the deployment is erroneous. This will be reviewed for a future release.
If GPS isn't available in a deployment (e.g., deployed indoors), then the deployment may fail with a server-side exception. It is important to note that the Arable system is based on availability of GPS in order to provide accurate locations on maps and in calculations at this stage. We are looking into how we can also operate better with adding the GPS coordinates via the API (and UI) if not reported by the Arable Mark 2.
The deployment of a Bridge is not immediately posted in the API (nor Web/Mobile UI) even if the Mark or Bridge device light UI shows it as connected. In addition, if an external sensor and/or Bridge are disconnected, the device(s) will not be removed from the API (or Web/Mobile UI), as they are kept accessible to show historical data. A separation of data storage and sensor/Bridge status will be done in a future release.
The summary endpoint that serves the Map UI in Arable Web & Mobile is today showing daily wind average. We will in a future release change this to show current-hour wind data instead.
While the API documentation at https://developer.arable.com/ is helpful, it is auto-generated and is missing a list of all available endpoints. We will add this into the Guide section in the future. In the meantime, if you need more help related to understanding the endpoints available on each API route, please contact Arable Customer Success via the Help icon in your app.