Embedded Systems Task

Your job is to find a real-world AI model failure within embedded engineering. You will give the prompt, context, files etc. and run a preset range of models against the problem.

If you have several model failures in mind, you can submit several ones (as many as you want).

We will use your submitted data to benchmark a wide range of models, and measure how well they perform at embedded engineering tasks.

Some examples:

Data sheet reading & conclusions

Embedded design survivals on worst case temperature, power or memory

Embedded architecture analysis

The high level areas:

1. Specifications: each value it took from the datasheet, with the condition it applies at.

2. Constraints: the figures it calculated from them, such as battery life in days, junction temperature, or RAM and flash use in bytes.

3. Architecture: the option those figures support, as a fixed label (A, B or C).

Example enviroments

• "Docker shell · pytest grader" for most tasks. The model gets a shell with Python for it to calculate
• "Docker shell · C harness with budgets" if the memory figure should come from a real build (size output, map file, stack usage).
• Internet off. Raise the timeout from 120 s to about 900 s: slower models need time to read the files and do the maths.

Checks

Break down the execution of the correct steps into "checks"

Each check is like a small reward to tell the AI model it is going in the right direction.

The model doesn't see the checks, but we use them to measure how "close" the model got to getting the answer right

A good task allows the AI to sometimes succeed, but mostly fail, and most runs land somewhere in between (passes some checks, but fails others)

Worked example (illustrative numbers)

A CR2032 sensor node must last 1,095 days. Sleep current is 1.5 µA typical at 25 °C, 4 µA max at 60 °C. Capacity is 220 mAh nominal, 154 mAh at -20 °C.

• Naive (wrong result): typical current, nominal capacity, 2,200 days. Picks "transmit every sample".
• Correct: worst case gives 962 days, so that option fails. Batching hourly gives 1,100 days, and the 1,920-byte buffer fits in SRAM. Picks "batch hourly".