Article Analysis · AI Engineering

How I Build AI Projects From Scratch

A repeatable, product-minded process for turning a blank repo into a real AI application — used building production AI/ML systems at Amazon, and demonstrated end-to-end by rebuilding the "ShelfScanner" bookshelf-recommendation app.

🎯 Portfolio + learning + real users 🧭 10-step workflow 🧪 Experiment-first 🤖 Spec-driven, not vibe-coding
3
Goals a project can serve
10
Steps in the workflow
2
Evaluation layers
100+
Commits in the rebuild

The Workflow at a Glance

Click any step to expand the reasoning behind it.

Core Arguments

01 · Motivation

Every project serves one of three goals

A project can teach a skill, demonstrate skills to employers, or — if real people use it — become experience or a product. These are different goals with different bars.

A learning project can be simple. The same project on a résumé needs robustness and visible good judgment — especially proof that the AI didn't just "vibe code some slop."

02 · Ideation

Problem first, technology second

Most people start with tech (RAG, agents, a new model) then hunt for a problem. Do the opposite: start with a problem you understand and work backwards to the technology.

Source problems from your own life, people around you, or a community (Reddit/Discord). Personal connection keeps you motivated and shows the enthusiasm and decision-making hiring managers actually look for.

03 · Scoping

Scope before you build

Turn a vague idea into a decision-ready problem: who it's for, constraints, input/output data, success metrics, latency, budget. A template captures this — most stays blank on purpose.

The question people skip: does AI even make sense here? If normal software solves it just as well, a model adds cost and unpredictability and makes you look inexperienced.

04 · Working with AI

Spec-driven, not outsourced thinking

Use coding assistants properly. The point of a portfolio project is to learn and demonstrate learning — outsourcing decisions to Claude means you can't defend them in an interview.

  • Set up docs directories + steering/rules files for the assistant
  • Install skills (e.g. Supabase) you'll rely on
  • Require a proposal before implementation — review it with a critical eye
  • Keep decisions in files, not chat history
05 · Foundations

Set up the boring parts

You will work with data, so you need somewhere to store it. The example uses Supabase (Postgres + file storage, generous free tier) matching scoping needs.

  • Create project, install CLI, grab API keys
  • Keep keys in a properly gitignored .env
  • Manage the Python env with a tool like uv
06 · The MVP

An MVP answers your biggest questions

An MVP is not a crappy version of the whole app. It's the smallest thing that answers the project's biggest unknowns — here, whether vision can read book spines and whether an LLM can produce useful recommendations.

ShelfScanner's first MVP was a command-line script: photo in → titles read → five recommendations out. That's it.

07 · Experiments

Treat it like a real experiment

Don't go by vibes. Log every model call: which model, which prompt, what it returned, latency, rough cost, and whether it failed.

Keep shelf-reading and recommendation stages separate — the best OCR model isn't necessarily the best recommender, and separating them lets you compare each problem cleanly and swap models later.

08 · Test Data

Start with a small labeled set

Begin with ~5 photos you already have. That can't prove the system works — it only answers "is this promising enough to continue?"

Write labels against a consistent rubric. Rules needn't be perfect, but they must be applied the same way, or grading becomes random and useless.

09 · Findings

What the MVP taught

  • Vision: Sonnet & Gemini Flash found essentially all readable books without hallucinating; smaller models missed titles and invented books.
  • Cost: less interesting than expected — good models sat inside budget.
  • Latency: a real problem — 10–12s just to read a shelf vs. a ~15s target for the whole app.
  • Recommendation: cheaper GPT-5.4 mini matched the author's choices better than pricier models, while the nicest-sounding explanations scored worse.
  • Privacy: source images carried GPS metadata that shouldn't be stored.
10 · Restraint

Leave out what the problem doesn't need

No points for cramming every technique into one repo. For ShelfScanner the author deliberately dropped:

  • No RAG — no retrieval problem; books come from the shelf.
  • No agents — the workflow is known ahead of time.
  • No fine-tuning — off-the-shelf models are good enough.
  • No logins, accounts, or purchase links in V1.
11 · Phases

Build riskiest parts first

Break work into phases that each leave something working end-to-end. Order matters more than the exact phases: tackle risky parts early enough that they can still change the design.

ShelfScanner: Phase 1 model router + automated quality check → Phase 2 smallest useful app (photo → titles) → Phase 3 preferences → …

Rough deadlines guard against scope creep. The rebuild: ~100 commits, ~10–15 hrs/week over 8 weeks — less than half spent actually coding, the rest studying.

12 · Monitoring

Log everything, improve over time

  • The same model-call records from experiments power a simple dashboard (model, latency, cost, prompt version, failures).
  • Optionally add alerts for unexpected behavior.
  • Turn repeated real-world failures into new eval-set cases.
  • Collect feedback signals (saves, "not for me") for product direction.
  • Add optimizations like caching only for a reason you can point to in an interview — never "because production apps have caches."
13 · Deployment & Career

Ship it, then present it

Vercel + GitHub gives near-click deployment with auto-deploys on change; a custom domain (shelfscanner.io) made it feel real.

The engineering is invisible from the outside, so a well-written README (trade-offs, functionality, architecture diagrams) is essential. Then get real users — even a few dozen teach you a lot and give interview stories. At scale, a real product can move from the "Projects" section to "Experience."

Evaluation: Two Layers of Correctness

Layer 1

Normal software tests

  • Test pieces with a known correct behavior
  • Valid uploads pass; bad inputs rejected; data attached to the right user
  • Full browser flow tests — parts can pass while the product breaks
  • Runs in CI/CD via GitHub Actions on every push/PR
Layer 2

AI evaluation set

  • Rerun on any change touching models, prompts, or preprocessing
  • Labeled photos + deliberately degraded versions (blur, glare, rotation, low-res)
  • Catches regressions and hallucinations — nothing crashes when behavior degrades
  • Compare prompt versions systematically — only possible with versioned prompts
Change typeDeterministic codeModel / prompt behavior
GPS strip from photosUnit / normal testNot relevant
Prompt tweak (vision)Not relevantEval set needed
Config changeSome coverageEval flagged a failure
Saved books ownershipUnit / normal testNot relevant

Key Takeaways

What to carry into your own project

  1. Decide why you're building — learning, demonstrating, or real users. Each raises the bar differently.
  2. Start from a problem you understand and work backwards to the technology.
  3. Scope before building — and always ask whether AI is even the right tool.
  4. Use spec-driven development so you make the decisions and can defend them.
  5. Make the MVP answer your biggest unknowns — not a flimsy copy of the whole app.
  6. Log every model call and separate stages so you can compare, monitor, and swap components.
  7. Build riskiest parts first and give each phase a rough deadline.
  8. Evaluate on two layers — deterministic tests and an AI eval set with versioned prompts.
  9. Monitor in production and let real failures feed the eval set.
  10. Ship, present well (README), and get real users — that turns a project into experience.

Don't copy ShelfScanner. Copy the process.

The project is a framework, not a template — the value comes from making your own decisions.

Deeper Detail

+The scoping template's full scope (what it eventually covers)

Product requirements, data, models, evaluation, infrastructure, security, deployment, monitoring and more — but most of it stays blank at the start. Early on you fill only what helps you understand the problem: user, constraints, input/output data, and success metrics. The rest gets filled in as evidence arrives from the MVP.

+Why the two AI stages are kept separate

There's no reason the model best at reading tiny text in an image is also the best at recommending books. Separating reading from recommending lets you compare each problem independently rather than treating a provider as one big package — and makes it clean to swap one model without touching the rest.

+The component menu you can draw from as a project grows
  • Prompt engineering — versioned prompts with a repeatable way to check if changes help.
  • Model selection — revisited as the product develops.
  • RAG — only if there's a real retrieval problem.
  • Agents — only if the model must choose tools/next steps; handle failures and loops.
  • The product around the AI — API, UI, auth, uploads, database.
  • Reliability — logging of errors, cost, and latency.
  • Advanced (fine-tuning) — only when it solves a found problem.

Not every project needs all of these — they're a menu, not a checklist.

+What "done" means for a change
  • You know what it was supposed to do.
  • Normal software checks pass.
  • The AI evaluation still looks good (if it can affect model behavior).
  • Documentation still describes the system you actually have.

Plus an assistant review of the change against the spec — another set of eyes. Most of this becomes boring once automated, which is the point.

+Honest caveat from the author

The finished ShelfScanner is a web app, not a native mobile app, and "not perfect." To hit the deadline the author still used AI more than a pure learning project would justify. The point was to demonstrate the process, with plenty left to optimize later.