AI training data and RLHF

AI training data and RLHF from employed software engineers

Empiric Infotech supplies AI labs and human data companies with employed engineers for RLHF, model evaluation and expert tasks. A professional writes the task, the correct answer and the rubric. The models are then tested against them, scored and ranked, with written feedback behind each score. Capacity switches on for a model release and steps back between releases, under one contract with one point of contact.

A service we provide to AI labs, human data companies and teams training or fine tuning their own models. Active in the micro1 Company Data Partnerships Program.

What we supply

AI programmes need tasks that separate one model from another, a professional's answer to measure against, and people who are ready when a new model lands.

Tasks a professional completes and a model fails

We turn real operational work into tasks that are hard for a frontier model: long horizons, ambiguous inputs, branching decisions and the judgement calls that only show up in production. A professional from that function writes the task, the correct answer and the rubric. A task earns its place only if it is hard because the work is difficult.

RLHF: scoring and ranking model answers

Engineers compare answers from several models on the same task, score them, rank them and write down why. That judgement is the human feedback RLHF depends on, and it comes from people who do this work for clients every day.

Capacity around each model release

Evaluation work comes in waves. Our engineers do it part time alongside client delivery, so capacity can switch on when a new model is released and step back between releases, with no full-time team for you to carry.

Employed engineers on one contract

Everyone on the work is an Empiric Infotech employee, checked and managed by us. You get one contract, one point of contact and one place to raise a quality or security issue.

The same workflow, across several models

One task run once against one model tells you almost nothing. We run the same workflow across multiple frontier models under the same conditions, so the differences that surface are differences in the models rather than differences in the prompt.

Scored per dimension, with the reasoning

Each run is graded against a rubric agreed with you before work starts, dimension by dimension, and every score carries written notes on what the model handled and where it went wrong. The output is not a single number. It is a map of which model is strong where, and why.

A sample of the work

One task from our evaluation work, with the task itself kept confidential. It shows what every task includes: a professional's answer, a rubric and real model runs scored against it.

What one task looks like in numbers

Reference files the model works from10
Rubric46 yes or no criteria
A professional's answer, marked with the rubric100%
Model A, average of 3 runs27.1%
Model B, average of 3 runs71%

In our first test of this task, every model scored 100%, so it could not tell them apart. We reworked it until a professional still scored 100% and the models did not.

How a rubric line reads

Criterion
The report gives the correct invoice total after the late credit note.
Passes when
It gives the total after the credit note, to the cent.
Does not pass when
It gives the total before the credit note, or gives no total.
Deduction
Points come off when the report names the supplier the brief asked to leave out.

Model names and task details stay confidential. The rubric lines above are an illustration written for this page.

What never leaves our side of the table

This work contributes operational expertise, not data about the people we work for. These are operating rules, applied before a workflow is written rather than checked afterwards.

Scrubbed or fictionalised from the start

Where a real workflow touches anything sensitive, it is rebuilt as a scrubbed or fictionalised version before any work begins. Not redacted afterwards.

Never client or customer material

No client, customer or employer confidential information. No proprietary material. Nothing covered by an NDA. This is the same commitment that governs our software delivery work.

No personal data, no credentials

No private or personal information about anyone, and no passwords, API keys or credentials in any prompt, file or environment.

Only tools we can properly grant

We work only in tools where the operator personally holds access and can safely extend that access to an AI system. Where a real process would run through a company system, we substitute a personal or fictionalised equivalent.

Never on unauthorised devices

Agent work does not run on any device we are not authorised to run it on.

The functions we can cover

Breadth from running these functions as a working software business, not from staffing up for a programme. We are also an official FlutterFlow Expert Agency Partner in the top 5% globally, listed in FlutterFlow's Experts directory, which is what betting early on a shift looks like when it pays off.

Software engineeringQA and testProduct and project deliveryCustomer supportSales and CRM operationsInternal operations

How an engagement runs

1

Scope

We agree the functions, the difficulty bar and the rubric dimensions with you before anything is built.

2

Build

A professional from that function writes the task, the correct answer and the rubric. The work is then scrubbed or fictionalised and checked against the data rules above.

3

Run

Each workflow is executed across the model set under matched conditions, so any difference that surfaces is attributable to the model rather than the setup.

4

Score and compare

Every run is graded on the agreed dimensions, then models are ranked against each other so the per dimension picture is legible.

5

Deliver

You receive the workflows, the runs, the scores and the comparison, in the structure your pipeline expects.

Questions partners ask

Do you ever use real client data?

No. Client confidentiality and intellectual property are non negotiable, and they do not bend for this programme. Where a workflow is drawn from real operational experience, it is rebuilt as a scrubbed or fictionalised version before the work starts, not cleaned up afterwards.

Do you supply RLHF work?

Yes. Our engineers score and rank model answers on the same task and write down the reasoning, which is the human feedback RLHF relies on. They can also write expert tasks with a correct answer and a rubric, for coding and for other functions we run.

Can your capacity follow our model release schedule?

Yes. Our engineers do this work part time alongside client delivery, so we can switch on for a release and step back afterwards. Tell us the release window and the volume you need, and we will confirm what we can staff before you commit.

Are your evaluators employees or contractors?

Employees of Empiric Infotech. They are checked and managed by us, under one contract with you and one point of contact.

Is this data annotation or labeling?

It is expert evaluation rather than bulk labeling. What we produce is expert authored tasks, comparative scoring across models and written feedback from a practitioner on what each model handled and where it failed. The unit is a task and a judgement, not a label.

What makes a workflow worth evaluating?

Difficulty that is real rather than artificial. The task has to reflect how the work is genuinely done, a professional in that field has to complete it with precision, and current frontier models have to struggle with it. A task that everything passes tells a programme nothing.

Who does the work?

The same practitioners who do the job for clients. The person who writes a QA workflow is a QA practitioner, and the person scoring a software engineering run is an engineer. That is the point: whoever judges the run has to know what good looks like in that function, or the score and the feedback are not worth much.

Which functions can you cover?

Software engineering, QA and test, product and project delivery, customer support, sales and CRM operations, and internal operations. The breadth comes from years of running these functions as a software delivery business, not from staffing up for a programme.

Can I join as an AI trainer, or can my company supply this to you?

This page describes a service we provide to companies. The people writing and scoring the work are our own employed practitioners, doing the job they do for clients, so the work stays in house. If you are looking for a role at Empiric Infotech, our open positions are at /career.

How do we start?

A short call to scope one batch. We would rather prove the work on a small, well defined set than talk about volume before you have seen the quality.

Related

Need engineers ready for your next model release?

If you are sourcing RLHF, expert tasks or comparative model evaluation, we would rather show you one batch than talk about volume. Tell us the functions, the rubric and the release window.

Start a conversation

SCOPE A BATCH

Tell us the functions you care about and the rubric you score on, and we will come back with what a first batch would look like.

Phone
0 / 1000
Attach a filePDF, DOC, or image. Maximum 10 MB.

We respond within one business day. Your details stay confidential.

Looking for a job? Use our careers form