Running Microsoft's BaseApp test buckets

Microsoft ships 34 Tests-* buckets inside the BC artifact, 32 of them non-empty, holding 40,530 [Test] methods — counted per bucket on the 28.1.49838.53507 platform artifact (#3409). They run through AL Runner as ordinary bundles, with no container.

Older notes in this repository say "about 40,550", and three of them stay that way on purpose: ParallelFanOut.cs and the two ParallelFanOut*TimeoutTests describe one specific past run ("a 40,550-test run down to 14,856"), and rewriting a recorded measurement to match a later count would falsify it. 40,530 is the counted figure for the artifact; 40,550 is what that run totalled. That makes them the largest supply of real, un-guessed work available: every failure is a concrete difference between the runner and what Microsoft's own tests expect.

The danger is the opposite of scarcity. A badly configured run produces thousands of plausible failures that are not defects at all, and an agent that clusters those files a stream of confident, wrong issues.

--test-data is mandatory

Not a refinement — a correctness precondition for the conclusions, not just the pass rate.

Measured on Tests-SMB (1,027 tests): 259 passing without test data, 595 with it. More importantly, in a full no-test-data run of 29,514 classified failures, the largest clusters were

2690  Order Nos. must have a value in Purchases & Payables Setup
2214  The General Posting Setup does not exist
2020  Order Nos. must have a value in Sales & Receivables Setup
1507  Invoice Nos. must have a value in Sales & Receivables Setup
1001  There is no Unit of Measure within the filter

Roughly 40% of all failures were missing setup data, not runner defects. Clustering that run and filing the top items would have produced a stream of issues describing nothing real.

With test data the same bucket's top clusters are genuine runner gaps — missing trigger dispatch, unsupported filter kinds, silently skipped handlers. Those are worth filing.

So: never file an issue from a run without --test-data. A no-test-data run is legitimate for measuring speed or for bisecting a regression, never for deciding what is broken.

And "the bc-linux container passes it" is not, by itself, a runner-gap verdict either. StefanMaron/MsDyn365Bc.On.Linux boots BC's Windows binaries with a startup hook that rewrites the methods that cannot work there — ALDatabase.ALSid answers a hashed SID, WindowsPrincipal.IsInRole is always true, RDLC rendering goes to a no-op CustomReportingServiceClient, encryption and Azure AD factories are bypassed. A cluster that passes on the container because of one of those is passing against the patch, and the runner has no BC behaviour there to copy; the 65-test ALSid cluster in #2312 was exactly that. Check a container-passing cluster against the patch table in docs/upstream-corpus-workflow.md § "How to find out whether a surface you care about is patched" before calling it a gap (#2314).

…and --test-data still gives a restored CRONUS, not a prepared one

The 40% above is the coarse form of a sharper fact. Microsoft does not restore the demo backup at all — it generates the test company from scratch, and --test-data presents the company as restored, not as Microsoft's tests were written against.

The recipe is known in full (#3429). microsoft/BCApps's build/scripts/ImportTestDataInBcContainer.ps1, for testType = "Legacy" — which is every BaseApp bucket — does this, in order:

1. install only the Base, TestFramework and LocalBaseExtensions app groups, plus DemoTool;
2. delete every existing company, then create one new, empty, non-evaluation company named CRONUS International Ltd.;
3. run codeunit 2 Company-Initialize, then open page 101900 "Demonstration Data Tool" and invoke Create Demo Data from Config, which runs codeunit 101899 against src/DemoTool/DemoDataConfig.xml with DataType = Extended;
4. install the remaining apps, after the demo data exists.

And step 3 cannot be replicated, because Microsoft does not publish the code that performs it. Measured, not assumed:

where	result
microsoft/BCApps src/DemoTool/	315 files — DemoDataConfig.xml, 95 png, 94 jpg, 32 gif, docs, spreadsheets. Zero .al files.
shipped Base Application 28.1	8,026 AL files, 1,691 codeunits, 2,610 pages. Zero objects in the 101000–101999 band. Zero files containing Create Demo Data from Config or Demonstration Data Tool.

Codeunit 2 Company-Initialize is in the shipped Base Application — confirmed twice, by an object-id scan of the package's AL sources and by name. Codeunit 101899 and page 101900 are in neither the artifact nor the public repository.

So "replicate Microsoft's data preparation" is not a task that can be assigned. Do not open an issue proposing it, and do not propose writing an importer for it — there is nothing to import. The config XML is public, the recipe is public, the executing code is Microsoft-internal.

What is reachable is exactly two things: codeunit 2, which ships and could in principle be run, and the demo backup, which is the output of that same pipeline shipped as data. --test-data already uses the second. So the gap is only where the shipped backup and a freshly generated company differ — a set of field-level differences, not a missing procedure.

A class of Microsoft-bucket failures is therefore neither a runner defect nor a BC divergence. It is a difference between two companies, and the only way to close a piece of it is to name the difference and write it down as a rule (see "Capturing a configuration difference" below).

This is deliberately not the priority (Stefan, resolving #2730): fix the clear runner failures first.

How much it is worth, measured. Do not go in expecting a large lever. Full Tests-ERM, both arms identical but for the flag:

	total	pass	fail	error
without --test-data-normalize-company	9,497	6,691	2,790	16
with it	9,497	6,709	2,772	16

+18 passing, +0.19 points, on the bucket where the one implemented rule should matter most. Both arms measured exactly the same 9,497 tests — identical key sets, nothing present in one and not the other — and all 18 flips are fail → pass with no regressions. 11 of the 18 are tests Microsoft itself never runs, so against their suite it is +7.

The mechanism is real and every flip went the way it was predicted to. The scale is what an earlier version of this section got wrong.

The denominator is also not 40,530. 12,018 of the 40,828 [Test] methods across the 32 buckets are listed in Microsoft's src/DisabledTests/ and are skipped by their own pipeline, so Microsoft runs 28,810 of them. A cluster made mostly of tests they disable is worth proportionally less. Correcting for it moves our headline from 59.2% to 60.3% — about one point, because we pass their disabled tests at 59.1% and their live ones at 66.9%.

One piece of the recipe now exists as an opt-in flag: --test-data-normalize-company. It rewrites named, measured fields of the restored company towards the DemoTool one; today the rule set is a single field, General Ledger Setup."Additional Reporting Currency" := ''.

It is off by default and every number below was measured without it. Turning it on changes which company the tests run against, so a normalized run's counts are not comparable with any recorded here — including the 259/595 Tests-SMB figures. The run says so itself: with the flag on, the summary prints every rule, what it changed and from which value, and names the rules that never fired.

Measured on this box (BC 28.1, --test-data, one bucket, everything else identical):

	without the flag	with it
Tests-SMB (1,028 discovered)	727 pass / 286 fail / 15 error	727 / 286 / 15 — no change
Tests-ERM Codeunit134157 in isolation	3 pass / 3 fail	6 pass / 0 fail

So the flag pays where the ACY is actually load-bearing and costs nothing where it is not. Tests-SMB has no ACY-sensitive assertion; #2730's clusters are in Tests-ERM. Do not read the flat Tests-SMB row as the flag not working — the run reported 1 of 1 row(s) changed (was 'EUR') in both cases.

To measure it across the surface, dispatch ms-surface.yml with normalize-company: true (#3450). It hands the switch down to ms-bucket.yml, which appends --test-data-normalize-company to the runner's argument array. The input defaults to false on every path — ms-surface.yml, ms-bucket.yml's two trigger blocks, and the nightly, which passes nothing and inherits it — so a run that leaves the field alone is still comparable with every number here. The nightly deliberately never turns it on: it is the trend line, and each point on it was measured un-normalized.

The triage rule

When a Microsoft bucket test fails, ask "is this a data-recipe failure?" before treating it as a runner defect. The tell is that the runner did the right thing for the company it was handed — the posting, the validation, the count are all correct given the data, and the expectation encodes a differently-prepared company.

Recipe failures:

• are expected, and stay failing until the recipe is replicated;
• must not be fixed by bending the runner to match the expectation;
• must not be filed as runner gaps;
• must not be classified expect-divergence — that mode means the runner intentionally answers differently from BC permanently (docs/expectations.md), and this is neither permanent nor a disagreement with BC. Calling it divergence records a fixable data gap as a settled decision.

Capturing a configuration difference

The triage rule above is entirely negative — do not file it, do not bend the runner, do not call it divergence. Here is the positive half, and it is the only way a piece of the recipe ever gets closed: name the difference and write it down as a rule.

Rules live in AlRunner/Infrastructure/TestDataNormalization.cs. One rule is one field write on one table's restored rows:

new CompanyNormalizationRule(
    TableId: 98,                                   // AL table id, never the backup's table NAME —
    TableName: "General Ledger Setup",             // the name varies by country layer, the id does not
    FieldName: "Additional Reporting Currency",
    TargetJson: "\"\"",                            // the target value as a JSON literal
    Why: "...")                                    // printed on every application; see below

Four things the file enforces, and that a PR adding a rule has to respect:

1. Bump RuleSetVersion. It feeds CacheIdentity(), which is folded into the install-baseline cache key. Skip it and a baseline captured under the old rule set gets restored into a run asking for the new one, silently, against un-normalized rows — the exact silent-wrong-answer class .claude/rules/loud-failures.md exists to prevent.
2. Why is not decoration. It is printed every time the rule fires, and it must say what Microsoft's company has, what the backup has instead, and which observable behaviour the difference changes. A reason not worth printing is not worth applying.
3. A rule that names a field the loaded rows do not carry THROWS. That is deliberate: the alternative is a run reporting "normalized" while nothing was normalized. If your rule throws, the rule is wrong, not the guard.
4. Field writes on existing rows only. Anything that is not one does not belong here, and this is where an eager agent goes wrong. #3429 names Global Dimension 2 and shortcut dimensions 3–6 as the obvious next candidates; they are master data. Changing Global Dimension 2 on a company that already has posted entries dimensioned by CUSTOMERGROUP is not a field write in BC, and the target value PROJECT does not exist as a Dimension in the restored company at all. Do not add them.

Evidence a new rule needs, before the PR. The same shape as the worked example below — run the affected bucket, or a single codeunit, both ways, and report both counts. --test-data-normalize-company is off by default precisely so that every recorded number stays comparable; a rule that ships without a measured before/after cannot be checked by anyone later.

Where to look for the difference itself. src/DemoTool/DemoDataConfig.xml in microsoft/BCApps is public, and it is the authoritative statement of what their generated company contains. It is the one half of the recipe we can read.

Read the right one. There are 25 files by that name in BCApps: src/DemoTool/DemoDataConfig.xml is W1, and the other 24 are country layers at src/GDL/<country>/DevBase/DemoTool/DemoDataConfig.xml (AU, BE, CA, CH, CZ, DACH, DE, DK, ES, FI, FR, GB, IN, IS, IT, MX, NA, NL, NO, NZ, RU, SE, AT, APAC). We run the W1 buckets, so W1 is the file that governs — a rule justified from a country layer's config is describing a company we do not build. The existing ACY rule's Why cites all 25 because it happens to hold in every one of them; that is a stronger claim than a rule normally needs, not the standard shape.

Compare it against what the backup actually holds, which the reader in AlRunner/TestDataProvisioner.cs can dump. The difference between those two is the whole search space.

The worked example, measured

Codeunit134157, three tests asserting a G/L Entry count, each off by exactly +1:

General Ledger Setup."Additional Reporting Currency"	result
EUR — what --test-data presents	3 failed / 3 passed
blank — what Microsoft's test database has	6 passed / 0 failed

Nothing else changed. HandleAddCurrResidualGLEntry opens with

if AddCurrencyCode = '' then exit;

so given an ACY, BC's own residual rule correctly adds a sixth G/L Entry, and the tests correctly report six where they expect five. The runner was posting correctly for the company it was handed. There is no runner defect anywhere in that chain.

The scale — a class, but a small one

#2730 records two more clusters from the same single setting: codeunit 134880's four Reverse… tests, and a 16-test exchange-rate cluster (There is no Detailed Cust. Ledg. Entry within the filter after report 596). #2833 is a fourth. One field of one setup table, four independent clusters, which is what makes this a recipe question rather than a handful of odd tests.

But measure before predicting. In the full-bucket run above, cu 134157 goes 3/6 → 6/6 and cu 134880 goes 22/28 → 26/28, both reproducing outside isolation — and the 16-test exchange-rate cluster did not move at all. The whole flag is worth 18 tests in a 9,497-test bucket. An earlier version of this section said "expect other prepared state to behave the same way", which primed agents to expect a large payoff; the honest version is that each difference is worth a handful of tests and the class is worth pursuing for correctness, not for the coverage number.

One thing this does NOT explain, and must not bury

Under ACY the runner's Additional-Currency amounts miss balance by 0.01 — debits 54,426.58 against payables -54,426.57. If the recipe blanks ACY, that divergence becomes unreachable in these tests rather than fixed. It may still be a real runner defect. Do not let "explained as a recipe gap" be read as "the arithmetic was fine".

Getting the sources

The buckets live in the platform artifact under Applications/BaseApp/Test/ as Tests-*.Source.zip, beside the .app files. AlRunner.Provisioning/ArtifactDownloader fetches them with an HTTP ranged read of the ZIP central directory rather than downloading the whole artifact — tools/DownloadArtifacts test-sources and test-data are the entry points. Each zip carries its own app.json and needs no edits; the $(app_*) version placeholders are fine.

--test-data additionally needs the demo backup (BusinessCentral-W1.bak, ~900 MB, from the sandbox artifact) at the selected build's artifact path, and the backup reader binary the runner looks for at ~/.cache/al-runner/bcbak/bcbak.

The configuration that must be exact

Get these wrong and the numbers mean nothing:

• Company is CRONUS International Ltd_ — trailing underscore, the SQL form, not a period. The run fails loudly listing both companies otherwise.
• Both package caches, --package-cache is repeatable: the platform apps and the test apps.
• Raise AL_RUNNER_EMIT_TIMEOUT_SEC well above its default for a large bucket. It is wall-clock, and a big bundle's emit takes minutes; under --jobs it is scaled per worker, but a single large bucket still needs headroom.
• Pass a private --cache <dir>. The shared cache is not keyed on the runner binary, so another process's build can silently change your results.

Sizing a run

Do not run all 33 at once to answer a question. Pick by what you are asking:

• A quick signal — Tests-SMB (1,027 tests, ~2 minutes warm with test data). Also the natural known-good baseline: 259 without test data, 595 with.
• A representative sample — Tests-ERM alone is 9,496 tests, about a quarter of the surface, and its cluster ranking has matched the full run's. Big enough to rank work, small enough to finish.
• The complete picture — all buckets, but expect hours and size the worker count from measured headroom. On a hosted runner this is a single dispatch of .github/workflows/ms-surface.yml (below); on a developer machine it competes with everything else for memory and has OOMed the box.

Memory, measured after the per-worker GC tuning: roughly 1.1 GB per worker without test data, ~2.3 GB with it (including its backup-reader sidecar). Derive the job count from free RAM, never hardcode it, and set MemoryHigh below MemoryMax so a cgroup throttles before the kernel's global OOM killer starts choosing victims elsewhere on the machine.

A single bundle cannot be split across workers, so the largest bucket sets the wall-clock floor however many workers you add.

Turning failures into issues

1. Cluster by normalized message — strip ids, numbers, quoted names, GUIDs and dates so one defect lands in one bucket. scripts/ carries the tooling; when attributing a failure to a bucket, key on the result header, not on the run's planning lines, or every failure is credited to whichever bundle was announced last.
2. Read the top cluster's stack, not just its message. Three shapes look identical and need different responses:
   • a real gap — the runner refuses or mishandles something BC supports;
   • a cascade — one early failure leaves state broken for the rest of the codeunit. In one measured case 46 of 47 failures were a cascade from a single test that renamed a row and died before restoring it. Fixing the first test fixes all 47, and filing 47 issues would have been noise;
   • a symptom — the failure is downstream of something else entirely. "Declared UI handler was not executed" turned out to have at least two unrelated causes.
3. Confirm against a clean cache before filing. A cache left inconsistent by a killed run silently cost 76% of passing tests once, and three commits were bisected before anyone tried a fresh cache. One re-run is cheaper than one wrong issue.
4. File with the measured count. "955 failures, 77% of this cluster, here is the sub-shape breakdown" is actionable. "Some tests fail" is not.

Known walls — do not refile these

Large clusters that are already understood, so a fresh run does not generate duplicates:

• Failures that vanish with --test-data (see above) are configuration, not defects.
• Failures that vanish when the company is prepared the way Microsoft's pipelines prepare it — the data-recipe class above — are not defects either. The ACY clusters (#2730, #2833, codeunit 134157, codeunit 134880's Reverse… four, the 16-test exchange-rate cluster) are the known instances. Do not refile them, do not bend the runner to them, and do not mark them expect-divergence.
• RunObject-only page actions are refused deliberately and loudly; supporting them is a feature, not a bug fix.
• The task scheduler, live external connections, report rendering, SMTP and HTTP egress are permanently out of scope — docs/scope.md is authoritative. A typed out-of-scope refusal with a named reason is the runner working correctly.

Before filing, search the issue queue for the area. A measured cluster is often already filed, sometimes several times over, and the whole set usually shares one root cause worth fixing together.

In CI

.github/workflows/ms-bucket.yml runs one bucket — or a list of them, sequentially in one job — on a hosted runner with the full configuration, workflow_dispatch only. It is a measurement job, not a gate: green means it produced a number, not that the suite passed. Prefer it over a developer machine when the machine is also doing something else.

.github/workflows/ms-surface.yml is the whole surface: one dispatch, one job, all 32 non-empty buckets in sequence, one combined total (#3409). Each bucket's numbers reach the job summary and the run's annotations as that bucket finishes, so a run that hits the 360-minute hosted ceiling still hands over everything that completed. It excludes Tests-TestLibraries (a library, zero tests) and Tests-Local (empty in the W1 artifact); the bucket list is held to the artifact's own inventory by AlRunner.Tests/MsSurfaceWorkflowTests.cs. For a subset, dispatch ms-bucket.yml with its buckets input rather than editing the surface list.

Sister material

autonomous-cycle — the unattended loop this feeds, and the gate that must pass before a cluster becomes an issue. project_ms_test_collections_run_recipe in session memory — the original recipe and its history.