AI training data and RLHF
AI training data and RLHF from employed software engineers
Empiric Infotech supplies AI labs and human data companies with employed engineers for RLHF, model evaluation and expert tasks. A professional writes the task, the correct answer and the rubric. The models are then tested against them, scored and ranked, with written feedback behind each score. Capacity switches on for a model release and steps back between releases, under one contract with one point of contact.
A service we provide to AI labs, human data companies and teams training or fine tuning their own models. Active in the micro1 Company Data Partnerships Program.
What we supply
AI programmes need tasks that separate one model from another, a professional's answer to measure against, and people who are ready when a new model lands.
Tasks a professional completes and a model fails
We turn real operational work into tasks that are hard for a frontier model: long horizons, ambiguous inputs, branching decisions and the judgement calls that only show up in production. A professional from that function writes the task, the correct answer and the rubric. A task earns its place only if it is hard because the work is difficult.
RLHF: scoring and ranking model answers
Engineers compare answers from several models on the same task, score them, rank them and write down why. That judgement is the human feedback RLHF depends on, and it comes from people who do this work for clients every day.
Capacity around each model release
Evaluation work comes in waves. Our engineers do it part time alongside client delivery, so capacity can switch on when a new model is released and step back between releases, with no full-time team for you to carry.
Employed engineers on one contract
Everyone on the work is an Empiric Infotech employee, checked and managed by us. You get one contract, one point of contact and one place to raise a quality or security issue.
The same workflow, across several models
One task run once against one model tells you almost nothing. We run the same workflow across multiple frontier models under the same conditions, so the differences that surface are differences in the models rather than differences in the prompt.
Scored per dimension, with the reasoning
Each run is graded against a rubric agreed with you before work starts, dimension by dimension, and every score carries written notes on what the model handled and where it went wrong. The output is not a single number. It is a map of which model is strong where, and why.
A sample of the work
One task from our evaluation work, with the task itself kept confidential. It shows what every task includes: a professional's answer, a rubric and real model runs scored against it.
What one task looks like in numbers
| Reference files the model works from | 10 |
| Rubric | 46 yes or no criteria |
| A professional's answer, marked with the rubric | 100% |
| Model A, average of 3 runs | 27.1% |
| Model B, average of 3 runs | 71% |
In our first test of this task, every model scored 100%, so it could not tell them apart. We reworked it until a professional still scored 100% and the models did not.
How a rubric line reads
- Criterion
- The report gives the correct invoice total after the late credit note.
- Passes when
- It gives the total after the credit note, to the cent.
- Does not pass when
- It gives the total before the credit note, or gives no total.
- Deduction
- Points come off when the report names the supplier the brief asked to leave out.
Model names and task details stay confidential. The rubric lines above are an illustration written for this page.
What never leaves our side of the table
This work contributes operational expertise, not data about the people we work for. These are operating rules, applied before a workflow is written rather than checked afterwards.
Scrubbed or fictionalised from the start
Where a real workflow touches anything sensitive, it is rebuilt as a scrubbed or fictionalised version before any work begins. Not redacted afterwards.
Never client or customer material
No client, customer or employer confidential information. No proprietary material. Nothing covered by an NDA. This is the same commitment that governs our software delivery work.
No personal data, no credentials
No private or personal information about anyone, and no passwords, API keys or credentials in any prompt, file or environment.
Only tools we can properly grant
We work only in tools where the operator personally holds access and can safely extend that access to an AI system. Where a real process would run through a company system, we substitute a personal or fictionalised equivalent.
Never on unauthorised devices
Agent work does not run on any device we are not authorised to run it on.
The functions we can cover
Breadth from running these functions as a working software business, not from staffing up for a programme. We are also an official FlutterFlow Expert Agency Partner in the top 5% globally, listed in FlutterFlow's Experts directory, which is what betting early on a shift looks like when it pays off.
How an engagement runs
Scope
We agree the functions, the difficulty bar and the rubric dimensions with you before anything is built.
Build
A professional from that function writes the task, the correct answer and the rubric. The work is then scrubbed or fictionalised and checked against the data rules above.
Run
Each workflow is executed across the model set under matched conditions, so any difference that surfaces is attributable to the model rather than the setup.
Score and compare
Every run is graded on the agreed dimensions, then models are ranked against each other so the per dimension picture is legible.
Deliver
You receive the workflows, the runs, the scores and the comparison, in the structure your pipeline expects.
Questions partners ask
Do you ever use real client data?
No. Client confidentiality and intellectual property are non negotiable, and they do not bend for this programme. Where a workflow is drawn from real operational experience, it is rebuilt as a scrubbed or fictionalised version before the work starts, not cleaned up afterwards.
Do you supply RLHF work?
Yes. Our engineers score and rank model answers on the same task and write down the reasoning, which is the human feedback RLHF relies on. They can also write expert tasks with a correct answer and a rubric, for coding and for other functions we run.
Can your capacity follow our model release schedule?
Yes. Our engineers do this work part time alongside client delivery, so we can switch on for a release and step back afterwards. Tell us the release window and the volume you need, and we will confirm what we can staff before you commit.
Are your evaluators employees or contractors?
Employees of Empiric Infotech. They are checked and managed by us, under one contract with you and one point of contact.
Is this data annotation or labeling?
It is expert evaluation rather than bulk labeling. What we produce is expert authored tasks, comparative scoring across models and written feedback from a practitioner on what each model handled and where it failed. The unit is a task and a judgement, not a label.
What makes a workflow worth evaluating?
Difficulty that is real rather than artificial. The task has to reflect how the work is genuinely done, a professional in that field has to complete it with precision, and current frontier models have to struggle with it. A task that everything passes tells a programme nothing.
Who does the work?
The same practitioners who do the job for clients. The person who writes a QA workflow is a QA practitioner, and the person scoring a software engineering run is an engineer. That is the point: whoever judges the run has to know what good looks like in that function, or the score and the feedback are not worth much.
Which functions can you cover?
Software engineering, QA and test, product and project delivery, customer support, sales and CRM operations, and internal operations. The breadth comes from years of running these functions as a software delivery business, not from staffing up for a programme.
Can I join as an AI trainer, or can my company supply this to you?
This page describes a service we provide to companies. The people writing and scoring the work are our own employed practitioners, doing the job they do for clients, so the work stays in house. If you are looking for a role at Empiric Infotech, our open positions are at /career.
How do we start?
A short call to scope one batch. We would rather prove the work on a small, well defined set than talk about volume before you have seen the quality.
Related
Need engineers ready for your next model release?
If you are sourcing RLHF, expert tasks or comparative model evaluation, we would rather show you one batch than talk about volume. Tell us the functions, the rubric and the release window.
Start a conversationSCOPE A BATCH
Tell us the functions you care about and the rubric you score on, and we will come back with what a first batch would look like.




