Skip to content

AI

  • in Tech, AI
  • 4 min read

Stop Over-Engineering Your LLM Apps in Production

Not too long ago, if you mentioned using LangChain in a production-ready enterprise system, seasoned developers would probably look at you with a mix of pity and concern. And rightly so. For a long time, the ecosystem felt like a playground for hobbyists and weekend hackers. Breaking changes were introduced on what felt like a daily basis, APIs shifted beneath your feet overnight, and the documentation was, to put it politely, a treasure hunt where the map was frequently written in an extinct language. And yet, there were tons of companies doing exactly this in a production-ready environment. Or should I say: "production"-"ready"...

But things have changed. A few months ago, the ecosystem finally hit its 1.0 milestone. The tools have officially matured, and they are ready for prime time.

However, "ready for use" doesn’t mean "use it everywhere." As the ecosystem grows up, we as developers need to grow up too. It’s time to talk about how to write this code safely, and more importantly, when to actually deploy it.

How: The Art of Dodging Outdated Code

Let’s start with the mechanics. Writing modern LangChain or LangGraph code requires a specific kind of discipline because the internet is currently gaslighting you.

Because the stable 1.0 architecture is still relatively young, the web is absolutely flooded with old tutorials, medium articles, and GitHub repos showcasing legacy syntax. If you blindly copy-paste code from a 2025 tutorial, your linter is going to throw a tantrum.

This brings us to a major caveat: Be extremely careful with AI coding assistants here. Tools like GitHub Copilot or ChatGPT are heavily trained on that massive ocean of outdated, pre-1.0 code. If you ask an LLM to generate a complex LangGraph workflow, it will confidently hallucinate deprecated methods and mismatched abstractions.

The strategy here is simple but demanding:

  • Make the official documentation your source of truth. Treat it like your primary manual.
  • Monitor your AI tools closely. Use them for boilerplate, but constantly double-check their outputs against the latest API references.
  • Avoid random internet tutorials unless you verify they were written after the stable release.

When: Resist the Urge to Over-Engineer

Now for the controversial part. Just because you can build something with LangChain doesn’t mean you should.

Right now, tech companies are suffering from a collective case of FOMO (Fear Of Missing Out). Teams are throwing LangChain and LangGraph at literally every problem, treating them like a magic fairy dust that makes software inherently better. Spoiler alert: it doesn’t.

Let's break down where these tools actually shine, and where they become a financial and performance liability.

The Sweet Spot: When LangGraph is a Lifesaver

Where LangGraph absolutely crushes it is in complex, multi-agent systems. If you are building a system where multiple specialized agents need to collaborate, loop back to previous steps, manage complex parallel workflows (concurrency), and maintain a robust state across a long-running conversation, writing that from scratch is a nightmare.

Yes, state management and multi-agent routing require a lot of coordination and inevitably consume more tokens to keep all agents aligned. But in this case, the overhead is justified. LangGraph handles the heavy lifting beautifully, cutting down on boilerplate and reducing the bugs that naturally creep into complex asynchronous systems. Here, you are paying a token premium for actual, necessary architectural heavy lifting.

The Trap: The Hidden Token Tax of Simple RAG

On the flip side, I constantly see developers pulling in the entire LangChain framework just to build a simple chatbot with a basic RAG (Retrieval-Augmented Generation) pipeline. This is where the "framework tax" hits you directly in the wallet.

When you use high-level abstractions, you often lose visibility into what is actually being sent over the wire. Many pre-built chains and agents come with hidden prompt wrappers, verbose formatting instructions, and aggressive history-management strategies running under the hood.

The Token Reality Check: A simple, natively written API call sends exactly what you tell it to send - nothing more, nothing less. LangChain’s abstract wrappers can easily bloat your context window with hundreds of hidden tokens per request just to format systemic instructions you didn't explicitly ask for.

If your app just takes a user query, fetches three relevant documents from a vector database, and stuffs them into a prompt template to send to the LLM, you do not need a framework. Pulling in a massive ecosystem for a simple API wrapper introduces unnecessary performance overhead, adds bloated dependencies, and silently inflates your LLM API bill. Writing native code using the raw LLM client gives you total control over every single token entering and leaving your system. It's faster, cleaner, and infinitely cheaper to run at scale.

The Bottom Line

Just keep in mind: LangChain and LangGraph obey the exact same rules as any other software library in history. They are architectural tools designed to solve specific problems, not a fashion statement. They aren't a cool pair of sneakers you wear just because everyone else is wearing them.

Every abstraction layer you add comes with a cost - both in code maintainability and literal token expenses. Use them when your system’s complexity demands them. Keep it simple and raw when it doesn’t. Your production budget - and your teammates who have to maintain your code - will thank you for it.

Share on Share on Share on

  • in Tech, AI
  • 6 min read

Creating a simple local RAG system

We'll build a simple RAG system using local only models. We will not use LangChain, which is introducing many bloated dependencies, is much slower than direct Transformers usage, is not error-free and its documentation is mostly misleading. We'll use only bare Transformers functions for that. As a vector database for storing our embeddings from document, we'll use Faiss, which is really efficient in similarity search. Note it sits in RAM, not on a disk and is very fast.

What is a RAG?

Retrieval-Augmented Generation (RAG) is an AI framework that improves Large Language Model (LLM) accuracy by retrieving data from external, trusted sources (documents, databases) rather than relying solely on training data. It enables up-to-date, specialized answers, reduces hallucinations, and avoids costly model retraining.

In simple words: it allows to have a LLM having a specialized knowledge without retraining it. We'll build here a simple version of it, allowing loading a single PDF files and then having a chat. We will use only local models, without using any cloud. This means few things: - it's completely free - it's completely private (no data exposing to internet) - it's weaker than cloud models.

Models of choice

It's up to you - and depends mostly on your hardware (GPU and its VRAM). I've used here google/gemma-2-9b-it as LLM and BAAI/bge-large-en-v1.5 for creating embeddings. It works without any issues on 12GB VRAM GPU - and it works with different languages. You can, for instance, have a source in Polish and ask questions in English (or vice-versa). Keep in mind in order to use models from HuggingFace, you have to have an account there and you have to accept the model's usage policy.

Some important parameters

When creating a RAG system, there're few parameters, that can have a big impact on how it's working. This includes: - LLM's temperature: it controls the randomness of LLM's output. The lower temperature, the more deterministic answers will be. - If LLM can do sampling: if sampling is set to off, LLM is using greedy sampling, so it just selects the tokens with the highest probability. If sampling is allowed, it will take one of the possible tokens. The choise is weighted, but it doesn't necessarily mean the highest probability will be chosen. - How many similar chunks to choose: during looking for similar chunks in the vector database, how many of them will be selected for the answer? In this example well working values are 3-5.

All in all, how you set them, depends mostly on the type of documents you want to work with: if it's some theory, reports, technicals, instructions, guides etc., then set it as I did below. If it's more loose texts, you may want to increase the temperature (let's say, up to 0.4) and turn sampling on. With even more informal texts you will also want to increase the number of similar chunks to be found (even above 10), but be careful, it will greatly increase the RAM usage.

Let's gather it up: 

EMBEDDING_MODEL = "BAAI/bge-large-en-v1.5"
LLM_MODEL = "google/gemma-2-9b-it"
LLM_TEMPERATURE = 0.1
LLM_DO_SAMPLE = False
SIMILAR_CHUNKS_COUNT = 3

Building a vector database

Let's start from creating our vector database, which is our special knowledge, created from given PDF file. We will read the file and split the text into smaller chunks (remembering the page number for each chunk, so we can give exact citations in our answers). The chunks will be converted into embeddings, using SentenceTransformer and our embedding model.

reader = pypdf.PdfReader(pdf_path)
full_text = ""
pages_meta = []  # A list to track which page each character comes from.
for i, page in enumerate(reader.pages):
    page_text = page.extract_text()
    if page_text:
        full_text += page_text
        # For each character on the page, store its page number and source file. This is a bit memory-intensive
        # but allows for accurate source tracking later.
        pages_meta.extend([{'page': i + 1, 'source': pdf_path}] * len(page_text))

Having the text extracted, we'll split the full text into smaller chunks:

chunks = simple_text_splitter(full_text, chunk_size=800, chunk_overlap=150)

We'll create a metadata for each chunk by finding the page number corresponding to the middle of the chunk:

chunk_metadatas = []
char_count = 0
for chunk in chunks:
    mid_point = char_count + len(chunk) // 2
    if mid_point < len(pages_meta):
        chunk_metadatas.append(pages_meta[mid_point])
    else: # A fallback for the very last chunk.
        chunk_metadatas.append(pages_meta[-1])
    char_count += 800 - 150 # Move character counter forward by (chunk_size - chunk_overlap).

Now we can create embeddings for each text chunk: 

embedding_model = SentenceTransformer(EMBEDDING_MODEL)
embeddings = embedding_model.encode(chunks, convert_to_tensor=True, show_progress_bar=True)
embeddings = embeddings.cpu().numpy().astype('float32') # FAISS requires float32 numpy arrays.

Normalize the embeddings to unit length. This is necessary for using the Inner Product (IP) as a measure of cosine similarity: 

faiss.normalize_L2(embeddings)

Let's populate the FAISS vector store: 

index = faiss.IndexFlatIP(embeddings.shape[1])
index.add(embeddings)  # Add the chunk embeddings to the index.

and pack all of this into a handy structure for later use: 

return {
    "index": index,
    "chunks": chunks,
    "metadatas": chunk_metadatas,
    "embedding_model": embedding_model
}

Feeding this structure into function below (as a db parameter), we can have a very simple and straightforward searching mechanism:

def search_vector_db(db, query, k):
    query_embedding = db["embedding_model"].encode([query], convert_to_tensor=True)
    query_embedding = query_embedding.cpu().numpy().astype('float32')
    faiss.normalize_L2(query_embedding)

    distances, indices = db["index"].search(query_embedding, k)
    retrieved_chunks = [db["chunks"][i] for i in indices[0]]
    retrieved_metadatas = [db["metadatas"][i] for i in indices[0]]

    return retrieved_chunks, retrieved_metadatas

query is a query. k is the number of chunks to be found.

Before we can use this for searching, we have to have a query first, so let's setup a simple chat with LLM.

Chat with LLM

Loading the LLM is pretty straightforward, if we remember the notes we stated in the beginning (the temperature etc.). We will load quite a big model, but quantize it to 4-bit precision, significantly reducing the size in RAM:

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True, 
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

tokenizer = AutoTokenizer.from_pretrained(LLM_MODEL)
model = AutoModelForCausalLM.from_pretrained(
    LLM_MODEL,
    quantization_config=bnb_config,
    device_map="auto",
    low_cpu_mem_usage=True
)
pipe = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    max_new_tokens=512,
    max_length=None,
    temperature=LLM_TEMPERATURE,
    repetition_penalty=1.1,
    do_sample=LLM_DO_SAMPLE,
    return_full_text=False
)
return pipe

Preparing the interactive loop to chat

Pretty much everything below will be closed inside the loop, so we can have a chat: 

while True:
    query = input("\nAsk a question (type 'exit' or 'quit' to quit): ")
    if query.lower() in ['exit', 'quit']:
        break

Searching the vector database

First thing to do after getting a query from a user, is to find relevant chunks of text, along with their page numbers, which we will use in sources citation.

context_chunks, context_metadatas = search_vector_db(db, query, k=SIMILAR_CHUNKS_COUNT)
context = "".join(context_chunks)

Having the relevant parts of the text, we have to prepare the LLM's part of the job.

Preparing the prompt template

This part will differ slightly depending on the LLM model used. Some models expect different prompt templates, regarding the user -> assistant loop: 

template = f"""<start_of_turn>user
You're a helpful assistant. Answer the question based only on the context below.
Answer using the same language the question was asked in.\n
Context:\n
{context}\n
Question: {query}<end_of_turn>
<start_of_turn>model
"""

If we don't formulate this properly, model can start hallucinating the dialog and start talking to itself. Note we have passed not the text, but our chunks that have been found after the user's question.

Finally we can pass it to LLM and get the answer.

result = llm_pipeline(template)
answer = result[0]['generated_text'].strip()
print(answer)

Sources citation

We can also quote the pages containing the relevant material in the PDF:

seen_pages = set() # Use a set to avoid printing duplicate page numbers.
for meta in context_metadatas:
    page_num = meta.get('page', 'N/A')
    if page_num not in seen_pages:
        print(f"  - Page: {page_num} (Source File: {meta.get('source')})")
        seen_pages.add(page_num)

Final thoughts

As you can see above, it's very simple, much simpler than one could think: it's not LLM looking for the answer. It's FAISS looking through all our chunks of text and finding the most suiting ones. LLM is given only those found ones and all it's doing is recapping the small portion of the text and formulating nice text. Simple as that.

Now, your task now is to put it into some UI, for example a simple Streamlit which is perfectly suiting this type of job. Just add some button to load the PDF, input and text - and voilà 😄

Share on Share on Share on

  • in Tech, AI
  • 3 min read

Diagramify - automatic diagram creation for Notion

I'm often writing descriptions and ideas on projects in Notion. And I thought: it'd be nice to have such system descriptions summarized in a diagram of proper kind, be it a flow diagram, decision diagram etc. And it'd be nice to have it done automagically.

So, I've built a simple Notion integration. It's made of 2 parts: a server, acting like a Notion MCP server, connecting to api.notion.com, and a client using this server. That way I could omit using a full-blown official Notion MCP server, requiring full OAuth authorization, playing with tokens etc. It's just a simple tool running on my own local machine, connecting directly to API.

How it works?

The server part is mimicking what official MCP server is doing, but it's adapted to my needs, like if it gets a page, which is long, it gets care of getting all the parts, etc. That way the client itself is much easier to write and maintain. Both server and client are written in Python. All you need to do is to create an internal integration on Notion Integrations page. From there, take your Notion token and put it in .env file: 

NOTION_TOKEN = "YOUR_NOTION_TOKEN"

Server

Server is using Starlette and MCP modules as its main core, using SSE Transport. And yes, it should be updated to use HTTP Stream Transport instead, but I've left that for some day. Now, the server has a number of tools the client can use. My list of tools is different from the original Notion MCP server, as it's suited for my task. For this simple integration it's: - list_accessible_pages: gets all the pages that were connected to our integration in Notion UI; connecting a page to an integration - get_notion_object: gets the object and its info. If it's a big object, get all the blocks. - upsert_mermaid_block: inserts or updates the Mermaid block.

Client

Now, how it's working? As the core functionality is automagic diagram insertion, I'm just using Gemini 2.5 Flash AI model to generate the diagram, feeding it with the whole text of the page. Inserted diagram is timestamped, so the client can know if the page was updated afterwards and needs refreshing of the diagram.

The exact flow of the client is: - get the list pages it has access to; - download whole page; - search for a signature with a timestamp; - if the signature doesn't exist, or the timestamp is older than the update date of the page: insert or update the mermaid block, the code itself comes from Gemini.

Simple as that.

There's also one thing to consider, while creating Mermaid diagrams with Gemini: while it's really good at it, it can sometimes do syntax errors, especially in complicated diagrams. That's why my prompt lists few rules.

Another thing is a hard limit of a single block length in Notion. It's only 2000 characters. It sounds as enough, but for large complicated diagrams it can be too small. Hence my prompt says exactly that and adds "if it's too long, simplify the diagram and omit less important elements".

Automating the run

I'm using Windows. For such tasks I like using the PM2. All it needs is creating a simple config file: 

module.exports = {
    apps : [
    {
        name: "notion-server",
        script: "notion_server.py",
        interpreter: "./PythonEnv/python.exe",
        autorestart: true,
        watch: false,
        env: {
            NODE_ENV: "production",
            PYTHONIOENCODING: "utf-8",
            PYTHONUTF8: "1"
        }
    },
    {
        name: "notion-worker",
        script: "notion_client_local.py",
        interpreter: "./PythonEnv/python.exe",
        autorestart: true,
        restart_delay: 300000,
        watch: false,
        env: {
            NODE_ENV: "production",
            PYTHONIOENCODING: "utf-8",
            PYTHONUTF8: "1"
        }
    }
  ]
}

With this config, it will take care of running the server and running the client every 5 minutes.

The sources

All the sources are available on GitHub, just keep in mind it's in Polish (both comments and Gemini's prompt), but I think you can easily get the grasp of what it's doing. The code is really simple.

Share on Share on Share on

  • in Tech, AI
  • 3 min read

Diagramify - automatic diagram creation for Notion

I'm often writing descriptions and ideas on projects in Notion. And I thought: it'd be nice to have such system descriptions summarized in a diagram of proper kind, be it a flow diagram, decision diagram etc. And it'd be nice to have it done automagically.

So, I've built a simple Notion integration. It's made of 2 parts: a server, acting like a Notion MCP server, connecting to api.notion.com, and a client using this server. That way I could omit using a full-blown official Notion MCP server, requiring full OAuth authorization, playing with tokens etc. It's just a simple tool running on my own local machine, connecting directly to API.

How it works?

The server part is mimicking what official MCP server is doing, but it's adapted to my needs, like if it gets a page, which is long, it gets care of getting all the parts, etc. That way the client itself is much easier to write and maintain. Both server and client are written in Python. All you need to do is to create an internal integration on Notion Integrations page. From there, take your Notion token and put it in .env file: 

NOTION_TOKEN = "YOUR_NOTION_TOKEN"

Server

Server is using Starlette and MCP modules as its main core, using SSE Transport. And yes, it should be updated to use HTTP Stream Transport instead, but I've left that for some day. Now, the server has a number of tools the client can use. My list of tools is different from the original Notion MCP server, as it's suited for my task. For this simple integration it's: - list_accessible_pages: gets all the pages that were connected to our integration in Notion UI; connecting a page to an integration - get_notion_object: gets the object and its info. If it's a big object, get all the blocks. - upsert_mermaid_block: inserts or updates the Mermaid block.

Client

Now, how it's working? As the core functionality is automagic diagram insertion, I'm just using Gemini 2.5 Flash AI model to generate the diagram, feeding it with the whole text of the page. Inserted diagram is timestamped, so the client can know if the page was updated afterwards and needs refreshing of the diagram.

The exact flow of the client is: - get the list pages it has access to; - download whole page; - search for a signature with a timestamp; - if the signature doesn't exist, or the timestamp is older than the update date of the page: insert or update the mermaid block, the code itself comes from Gemini.

Simple as that.

There's also one thing to consider, while creating Mermaid diagrams with Gemini: while it's really good at it, it can sometimes do syntax errors, especially in complicated diagrams. That's why my prompt lists few rules.

Another thing is a hard limit of a single block length in Notion. It's only 2000 characters. It sounds as enough, but for large complicated diagrams it can be too small. Hence my prompt says exactly that and adds "if it's too long, simplify the diagram and omit less important elements".

Automating the run

I'm using Windows. For such tasks I like using the PM2. All it needs is creating a simple config file: 

module.exports = {
    apps : [
    {
        name: "notion-server",
        script: "notion_server.py",
        interpreter: "./PythonEnv/python.exe",
        autorestart: true,
        watch: false,
        env: {
            NODE_ENV: "production",
            PYTHONIOENCODING: "utf-8",
            PYTHONUTF8: "1"
        }
    },
    {
        name: "notion-worker",
        script: "notion_client_local.py",
        interpreter: "./PythonEnv/python.exe",
        autorestart: true,
        restart_delay: 300000,
        watch: false,
        env: {
            NODE_ENV: "production",
            PYTHONIOENCODING: "utf-8",
            PYTHONUTF8: "1"
        }
    }
  ]
}

With this config, it will take care of running the server and running the client every 5 minutes.

The sources

All the sources are available on GitHub, just keep in mind it's in Polish (both comments and Gemini's prompt), but I think you can easily get the grasp of what it's doing. The code is really simple.

Share on Share on Share on

  • in Tech, AI
  • 9 min read

AI is being used in recruitment widely and everybody knows that. That led me to think: what if we do it the opposite way? Give a user the power of AI in his job search? Why and what for? For two reasons: first and obvious: let him find a better suited job. But there's also another thing: by studying the results we could see what in the user's profile made a given job offer to be suitable or not? Maybe there can be something changed in the profile? Because... that's also probably the same thing the recruitment's AI system is observing. So, in this way, user could make his profile better.

Now, the whole system I've made consists of 4 stages: web crawling, offer analysis, profile analysis and finally job matching. The final output is a proposal of a decision (apply, reject, maybe), summary of pros and cons and some thoughts and reasoning.

img

Web crawling is a huge thing and I won't discuss it here, as it's way too big for this article. I'll just mention it requires analysing the pages, deciding if it's even related, if it's a job offer or maybe a job offers list. It should analyze the links inside and find consecutive subpages. The output should be pairs of: url of an offer and its full text, without any HTML tags etc., just a pure text. All the other elements of this system we'll discuss below.

Technical stuff

Before we start diving into details of each of the stages, let's talk for a moment about technology used here. I've decided to make it as simple as possible - yet very easy to understand - and easy to run on user's home machine. That's why for provider of LLM models I chose Ollama, very handy system for self hosted LLMs. Coding will be done in Python (I'm using Python 3.12) using only elementary additional packages, like json, PdfReader, tqdm, requests and ollama. For communicating with LLMs through Ollama I've made a simple class with two types of clients: the one using ollama implementation and the one communicating through HTTP JSON API. You can see this little library on GitHub. It also contains few examples on how to use it in various scenarios. In this project we will use the structured JSON way.

Some general thoughts on how to utilize AI in similar jobs

The key is a very precise system prompt, defining how the model should act and what it should produce as an output. Something like "You are a job offer analyzer ... Return the output as JSON with the following fields ... do not add any explanations. Output only JSON ...". Also setting a model's temperature to 0 helps getting highly consistent, deterministic, and focused output. Another thing is to granulate the overall job to as small steps as possible. This is what we can see on above diagram - do not try to do few steps in one go, give model a single and precise job to do.

Offer analysis

The input to this stage is a pair of info: an url of an offer and its full text, stripped from any HTML tags etc - just a text. As an output we want a structured info, containing all the relevant info, like title, salary, type of job, requirements, nice-to-have etc. Let's see an example:

{
    "url": "https://great.company.com/job-offer",
    "company": "Great Company",
    "title": "Tech Lead",
    "location": "100% remote",
    "remote": true,
    "seniority": "lead",
    "salary": "",
    "description": "Provide technical leadership to the delivery team, be accountable for delivering defined feature sets, design and develop components within the data and analytics layer of an investment research platform, co-create system architecture and technology standards, ensure solution quality through code reviews, mentoring, and oversight of engineering practices, support the Product Owner and work closely with the team in backlog planning and execution, actively contribute to the development of analytical tools for investment analysts, participate in R&D work related to future iterations of the platform.",
    "responsibilities": [],
    "requirements": [
        "proven experience in providing technical leadership and acting as a Tech Lead in enterprise scale projects",
        "expert knowledge of agile software delivery and DevOps across the SDLC",
        "strong mentoring and coaching skills, including implementing engineering, architecture, and testing best practices",
        "experience in initiating and driving continuous improvement initiatives",
        "ability to work closely with Product Owners, stakeholders, and business users",
        "English proficiency at a minimum B2+ level"
    ],
    "nice_to_have": [],
    "technologies": [
        "agile software delivery",
        "DevOps",
        "SDLC"
    ],
    "offer": [],
    "language": "en"
}

Most important thing (as in whole this system) is a system prompt, making sure AI will return a precise output, formatted as we need:

system_prompt = """
You are a job offer analyzer. 
Extract structured information from the job description text. 
Return the output as JSON with the following fields:

- url: URL of the job offer
- company: infer company name if possible from the text or URL
- title: job title
- location: city / country / "remote"
- remote: true / false / "unknown"
- employment_type: full-time / contract / internship / unknown
- seniority: junior / mid / senior / lead / manager / unknown
- salary: salary range if specified
- description: short summary in your own words
- responsibilities: list of main responsibilities
- requirements: list of key requirements
- nice_to_have: list of additional "nice to have" skills
- technologies: programming languages, frameworks, tools
- offer: list of benefits, if available
- language: language of the job offer (en/pl)

Do not add any explanations. Output only JSON.
Never invent jobs that are not clearly present.
Never hallucinate technologies.
If a field is missing, use empty string, empty array, or "unknown".
"""

Then, the function itself is really simple if we'll use the ollama library I've quoted above:

def extract_job_offer(url, text):
    llm_client = LLMClientOllama()
    llm_client.set_model("qwen3:14b")
    llm_client.set_temperature(0)
    llm_client.set_json_format(True)

    data, role = llm_client.call_llm(f"URL: {url}\n\nText:\n{text}", system_prompt)
    return data

call_llm method from LLMClientOllama class will take care of proper JSON payload sending, receiving and extracting from the response.

💡
For all the structuring/extraction jobs I've chosen the QWEN3 model, but you can check other models. Specifically, QWEN3 14B is running smoothly on nVidia with 12GB of VRAM.

User's profile creation from CV

Before we can match the offer, we need to have a second side - the user's profile. Again, it will be a structured JSON. We could do it manually, but we'll make a PDF extractor, as headhunting systems are doing. That can give us a feedback on how well our CV is composed.

First step is extracting all the text from CV. This already gives a hint: DO NEVER send PDFs made of graphics (scanning or other composition tools) - it has to be a real text (not rendered). For this task we'll use PdfReader class from pypdf package.

from pypdf import PdfReader

def extract_text_from_pdf(pdf_path: str) -> str:
    reader = PdfReader(pdf_path)
    pages = []

    for page in reader.pages:
        text = page.extract_text()
        if text:
            pages.append(text)

    return "\n".join(pages)

Then we prepare the system prompt:

CANDIDATE_SYSTEM_PROMPT = """
You are an expert technical recruiter and career analyst.

Your task is to analyze a CV (resume) and extract a structured candidate profile.

Rules:
- Output ONLY valid JSON
- Do NOT include explanations, markdown, comments or prose
- If some information is missing, infer conservatively or use null
- Normalize names (e.g. "C plus plus" → "C++")
- Seniority must be one of:
  ["junior", "mid", "senior", "lead", "staff", "principal", "staff / principal / lead", "manager", "director", "cto", "ceo", "unknown"]

The JSON schema MUST match exactly:

{
  "seniority": string,
  "years_of_experience": number,
  "primary_roles": string[],
  "core_languages": string[],
  "secondary_languages": string[],
  "domains": string[],
  "leadership": {
    "people_management": boolean,
    "tech_lead": boolean,
    "scrum_master": boolean
  },
  "cloud": string[],
  "devops": string[],
  "frontend_level": string,
  "remote_preference": boolean,
  "languages_spoken": { "pl": string, "en": string },
  "job_preferences": {
    "roles_to_avoid": string[],
    "preferred_roles": string[]
  }
}

Think carefully. This profile will be used for automated job matching.
"""

and a small function preparing a user prompt:

def build_candidate_prompt(cv_text: str) -> str:
    return f"""
Analyze the following CV and extract the candidate profile.

CV TEXT:
----------------
{cv_text}
----------------
"""

Now we're ready to build a profile from CV:

def build_candidate_profile_from_cv(pdf_path: str) -> dict:
    cv_text = extract_text_from_pdf(pdf_path)

    llm_client = LLMClientOllama()
    llm_client.set_model("qwen3:14b")
    llm_client.set_temperature(0.2)
    llm_client.set_json_format(True)

    profile, llm_role = llm_client.call_llm(build_candidate_prompt(cv_text), CANDIDATE_SYSTEM_PROMPT)
    return profile

Of course the resulting JSON data can be modified to tweak it, but also it could be used to verify if maybe something should be added to the original CV instead.

Algorithmic match

It would be tempting to do all matching job using AI, but there're at least two points against it:

  • every AI call takes time - and if we can avoid it with some obvious rejects, it's a plus;

  • collecting some info, categorizing it etc. will be done better in simple code, as AI may sometimes hallucinate things.

That's why as a first step in job matching we'll perform some algorithmic data collection and first decision.

What you can do in this step, of course depends on the kind of a job, but for software developers you could score things like tech stack, seniority, domains, leadership duties, and general logistics. Let's see an example:

def normalize_token(s: str) -> str:
    return s.lower().replace(" ", "").replace(".", "")

def extract_technologies_from_offer(offer) -> set[str]:
    required_text = " ".join(offer.get("requirements", [])).lower()
    optional_text = " ".join(offer.get("nice_to_have", [])).lower()

    known_tech = {
        "c++", "c#", "python", "java", "javascript", "typescript",
        "golang", "rust", "swift", "kotlin", "scala",
        "node.js", "react", "angular", "docker", "kubernetes",
        "aws", "azure", "gcp", "qt", ".net", "opengl", "vulkan", "webgl", "webassembly",
        "postgresql", "mysql", "mongodb", "redis", "elasticsearch", "postgres", "mongo",
        "gitlab", "git", "GitLab CI", "ci/cd", "github actions", "github"
    }

    found_required = set()
    found_optional = set()
    for tech in known_tech:
        if tech in required_text:
            found_required.add(tech)
        if tech in optional_text:
            found_optional.add(tech)

    return found_required, found_optional

def score_tech_stack(profile, offer):
    offer_required_tech, offer_optional_tech = extract_technologies_from_offer(offer)

    profile_tech = {
        normalize_token(t)
        for t in profile["core_languages"] + profile["secondary_languages"]
    }

    strengths = []
    gaps = []
    score = 0

    for tech in offer_required_tech:
        if normalize_token(tech) in profile_tech:
            strengths.append(tech)
            score += 10
        else:
            gaps.append(tech)
            score -= 5

    for tech in offer_optional_tech:
        if normalize_token(tech) in profile_tech:
            strengths.append(tech)
            score += 5

    return max(min(score, 30), 0), strengths, gaps

Whatever you do, in the end of this step you should have something like this as a result:

{
    "decision": "reject",
    "score": 50,
    "strengths": [
        "java",
        "python",
        "c++"
    ],
    "gaps": [
        "javascript"
    ]
}

You should set the overall score levels for taking a decision, depending on how you've set scoring. Set 3 levels of decision: apply / maybe / reject - and filter out the rejected ones before passing the offers to the next step, which is AI matching.

AI match

Now it's time for the last step - the key one. Its input would be an offer, user's profile and algorithmic matching results, so it can learn from it.

Let's start from a system prompt:

REVIEW_SYSTEM_PROMPT = """
You are a senior technical recruiter and staff-level software engineer.

Your task is to evaluate whether this job offer is worth applying to
for experienced software engineer with attached profile.

You MUST be critical and skeptical.
Reject roles that are:
- execution-only
- lacking ownership or technical impact

Return ONLY valid JSON.
Do NOT include markdown.
Do NOT include explanations outside JSON.

JSON schema:

{
  "final_verdict": "apply" | "maybe" | "reject",
  "confidence": 0-100,
  "key_reasons": [string],
  "risks": [string],
  "positive_signals": [string],
  "summary": string
}
"""

Of course it should be altered to suit your needs.

Next, let's prepare a user prompt:

def prepare_llm_input(profile: dict, offer: dict, match: dict) -> dict:
    """
    Builds a clean, stable input structure for LLM evaluation.
    No formatting, no text generation.
    """

    return {
        "candidate": {
            "seniority": profile.get("seniority"),
            "years_of_experience": profile.get("years_of_experience"),
            "core_stack": profile.get("core_languages"),
            "secondary_stack": profile.get("secondary_languages"),
            "domains": profile.get("domains"),
            "leadership": profile.get("leadership"),
            "preferences": {
                "remote": profile.get("remote_preference"),
                "preferred_roles": profile.get("preferred_roles"),
                "roles_to_avoid": profile.get("roles_to_avoid"),
            }
        },

        "job": {
            "title": offer.get("title"),
            "location": offer.get("location"),
            "responsibilities": offer.get("responsibilities", [])[:10],
            "requirements": offer.get("requirements", [])[:10],
            "nice_to_have": offer.get("nice_to_have", [])[:5]
        },

        "algorithmic_assessment": {
            "score": match.get("score"),
            "decision": match.get("decision"),
            "strengths": match.get("strengths", []),
            "gaps": match.get("gaps", []),
            "red_flags": match.get("red_flags", [])
        }
    }

def build_llm_prompt(llm_input: dict) -> str:
    """
    Converts structured LLM input into a readable prompt.
    """

    return f"""
Candidate profile:
{json.dumps(llm_input["candidate"], indent=2, ensure_ascii=False)}

Job offer:
{json.dumps(llm_input["job"], indent=2, ensure_ascii=False)}

Algorithmic assessment:
{json.dumps(llm_input["algorithmic_assessment"], indent=2, ensure_ascii=False)}

Evaluate realistically whether applying makes sense.
"""

Now we can call it:

def review_offer_llm(profile: dict, offer: dict, match: dict) -> dict:

    llm_data = prepare_llm_input(profile, offer, match)
    llm_input = build_llm_prompt(llm_data)

    llm_client = LLMClientOllama()
    llm_client.set_model("qwen3:14b")
    llm_client.set_temperature(0)
    llm_client.set_json_format(True)

    data, role = llm_client.call_llm(llm_input, REVIEW_SYSTEM_PROMPT)

    return data

For this last step you could experiment with various models, as they can give slightly different reasoning. While the overall results will be probably similar, the reasoning can be very helpful for user can react and either take his action or maybe update his/her CV according to those results.

Let's see an example output of this step:

{
    "final_verdict": "reject",
    "confidence": 85,
    "key_reasons": [
        "Role lacks technical ownership and leadership responsibilities",
        "Candidate's seniority (manager) far exceeds job requirements",
        "Focus on code evaluation rather than system design/architecture",
        "Part-time hourly contract misaligned with candidate's experience level"
    ],
    "risks": [
        "Underutilization of candidate's leadership and technical expertise",
        "Potential for role to be perceived as junior-level despite candidate's seniority",
        "Mismatch between compensation structure (hourly) and candidate's career stage"
    ],
    "positive_signals": [
        "Remote work flexibility",
        "Opportunity to work with AI systems",
        "Python/C++ stack alignment"
    ],
    "summary": "While the technical stack aligns, the role's responsibilities and compensation structure are fundamentally misaligned with a senior manager's experience and career expectations. The position offers limited technical impact and leadership opportunities, making it unsuitable for someone with 25 years of experience in complex domains like medical devices and embedded systems."
}

Now it would be handy to render the results into some HTML page report. Just prepare some template and replace text with fields from our JSON data.

Share on Share on Share on