Advanced Scalable Architecture

01 What is transaction management in Spring Boot?

Solution and explanation

Solution:

Consider a banking transfer.

Account A:

balance = 10000

Account B:

balance = 2000

Customer transfers:

5000

The operation has two important database changes:

Account A:
10000 -> 5000

Account B:
2000 -> 7000

Both must succeed together.

If Account A is debited but Account B is not credited, the database becomes inconsistent.

A transaction groups the database changes.

If everything succeeds:

COMMIT

If a database operation fails:

ROLLBACK

Important:

A local database transaction cannot automatically roll back an external bank transaction that already succeeded.

Example:

Database transaction
|
+-- debit record
|
+-- external bank API
|
+-- SUCCESS
|
+-- database failure

A database ROLLBACK cannot undo the external bank's successful operation.

That situation requires idempotency, status reconciliation, callback handling and potentially a separate reversal/refund process.

Run this example

        @Service
        public class TransferService {

            private final AccountRepository accountRepository;
            private final TransferRepository transferRepository;

            @Transactional
            public void transfer(
                    Long fromAccount,
                    Long toAccount,
                    BigDecimal amount) {

                Account source =
                    accountRepository
                        .findByIdForUpdate(fromAccount)
                        .orElseThrow();

                Account target =
                    accountRepository
                        .findByIdForUpdate(toAccount)
                        .orElseThrow();

                if (source.getBalance().compareTo(amount) < 0) {
                    throw new InsufficientFundsException();
                }

                source.debit(amount);
                target.credit(amount);

                accountRepository.save(source);
                accountRepository.save(target);

                transferRepository.save(
                    Transfer.completed(
                        fromAccount,
                        toAccount,
                        amount
                    )
                );
            }
        }

Hands-on challenge

Implement an Account A -> Account B transfer and demonstrate rollback when the second database operation fails.

Quick check

Which statement is correct?

02 How do you handle two simultaneous withdrawals?

Solution and explanation

Solution:

Suppose bank account balance:

Rs 5,000

At the same time two requests arrive.

Request A:

withdraw Rs 5,000

Request B:

withdraw Rs 10,000

Without concurrency protection, both requests may read:

balance = 5000

Both may incorrectly believe enough money exists.

This is a race condition.

The database must serialize the balance decision.

One approach is pessimistic locking.

Request A locks the account row.

Request B waits.

Request A checks:

5000 >= 5000

Yes.

New balance:

0

Request A commits.

Request B obtains the lock.

It now reads:

balance = 0

Then:

0 < 10000

Therefore:

INSUFFICIENT_FUNDS

Only one operation succeeds.

The exact concurrency mechanism should match the database and transaction design.
The critical rule is that the balance check and balance update must occur under the same protected transaction.

Run this example

        @Lock(LockModeType.PESSIMISTIC_WRITE)
        @Query("""
            SELECT a
            FROM Account a
            WHERE a.id = :id
        """)
        Optional<Account> findByIdForUpdate(
            @Param("id") Long id
        );

        @Transactional
        public WithdrawalResult withdraw(
                Long accountId,
                BigDecimal amount) {

            Account account =
                repository.findByIdForUpdate(accountId)
                    .orElseThrow();

            if (account.getBalance()
                    .compareTo(amount) < 0) {

                return WithdrawalResult.failed(
                    "INSUFFICIENT_FUNDS"
                );
            }

            account.debit(amount);

            repository.save(account);

            return WithdrawalResult.success();
        }

Hands-on challenge

Execute two concurrent withdrawal requests against a Rs 5,000 account and verify that the balance never becomes negative.

Quick check

Which statement is correct?

03 How do you handle millions of API requests?

Solution and explanation

Solution:

Suppose a job portal receives:

1,000,000 requests per day.

The first question should not be:

"Which technology should we add?"

First identify the bottleneck.

Possible bottlenecks:

Database CPU
Slow SQL
Missing indexes
Too many database connections
Large responses
Expensive search
External API latency
Application CPU
Memory pressure

Then scale the appropriate layer.

Example architecture:

Users

CDN

Load Balancer

API Gateway

Multiple Spring Boot instances

Redis

Database / Search

Slow operations:

Spring Boot

Kafka / Queue

Background workers

For job search:

Use proper indexes or a dedicated search engine.

For repeated reads:

Use Redis.

For large result sets:

Use pagination.

For expensive background operations:

Use queues/Kafka.

For abusive clients:

Use rate limiting.

Scaling should be measured through metrics rather than adding infrastructure blindly.

Run this example

        Client
           |
        Load Balancer
           |
        +-----------------------+
        | Spring Boot Instance 1|
        | Spring Boot Instance 2|
        | Spring Boot Instance 3|
        | Spring Boot Instance 4|
        +-----------------------+
                  |
              Redis Cache
                  |
           +------+------+
           |             |
        Database      Search
           |
        Read replicas

Hands-on challenge

Measure an API bottleneck and apply caching, indexing, horizontal scaling and asynchronous processing where required.

Quick check

Which statement is correct?

04 How do you prevent duplicate API requests?

Solution and explanation

Solution:

Suppose a candidate clicks:

Apply Now

The browser sends:

POST /api/jobs/500/applications

Network delay occurs.

The user clicks again.

Two requests arrive:

Request A
Request B

Without protection:

application #100
application #101

Both may be created.

The business rule says:

One candidate can apply to one job only once.

Use multiple protection layers.

Application request:

Idempotency-Key: APPLY-USER-100-JOB-500

Database:

UNIQUE(candidate_id, job_id)

The service transaction checks and creates the application.

The database constraint is the final protection against concurrent requests.

Application-level checks alone are not enough because two requests can pass the check simultaneously.

Run this example

        @PostMapping("/jobs/{jobId}/applications")
        public ApplicationDto apply(
                @PathVariable Long jobId,
                @RequestHeader("Idempotency-Key")
                String idempotencyKey,
                Authentication authentication) {

            return applicationService.apply(
                jobId,
                authentication.getName(),
                idempotencyKey
            );
        }
        
        CREATE UNIQUE INDEX
        uk_candidate_job
        ON applications(candidate_id, job_id);

Hands-on challenge

Send the same application request concurrently and prove only one application is created.

Quick check

Which statement is correct?

05 How do you design a scalable job search system?

Solution and explanation

Solution:

A relational database can remain the authoritative source for jobs.

But searching millions of job documents with:

title skills company location experience salary industry employment type

can become expensive.

A search index such as Elasticsearch or OpenSearch can provide:

Full-text search
Skill matching
Location filters
Salary filters
Facets
Relevance
Pagination

Architecture:

Admin/Recruiter

Job Service

PostgreSQL

Outbox/Event

Kafka

Search Index

Search API

The important rule is:

Search index is not automatically the business source of truth.

PostgreSQL can remain authoritative.

When a job changes:

Database transaction records the job change.

An event is published.

Search indexing consumes the event.

If indexing fails, the event can be retried/reconciled instead of losing the business record.

Run this example

        Job Service
            |
            +---- PostgreSQL
            |
            +---- Outbox Event
                      |
                      v
                    Kafka
                      |
                      v
               Search Indexer
                      |
                      v
              Elasticsearch/OpenSearch

        GET /api/search/jobs?q=spring+boot&location=pune

Hands-on challenge

Create a job search index containing title, skills, location, experience and salary filters.

Quick check

Which statement is correct?

06 How do you design a production job application system?

Solution and explanation

Solution:

Consider:

POST /jobs/500/applications

The application service should perform:

Authenticate candidate.
Load job.
Verify job is open.
Verify candidate eligibility.
Check duplicate application.
Create application.
Create initial status.
Create audit record.
Commit transaction.
Publish ApplicationSubmitted event.
Notification service sends email.
Analytics service processes event.

Database example:

applications

id candidate_id job_id status applied_at

application_status_history

id application_id old_status new_status changed_by changed_at

The status history is important because the current status alone does not explain how the application reached that state.

Example:

APPLIED

SCREENING

SHORTLISTED

INTERVIEW

SELECTED

Every transition should be validated.

A candidate should not normally jump from APPLIED directly to an arbitrary invalid state unless the business rules permit it.

Run this example

        @Transactional
        public ApplicationDto apply(
                Long candidateId,
                Long jobId
        ) {

            Job job = jobRepository
                .findById(jobId)
                .orElseThrow();

            if (!job.isOpen()) {
                throw new JobClosedException();
            }

            if (applicationRepository
                .existsByCandidateIdAndJobId(
                    candidateId,
                    jobId
                )) {

                throw new DuplicateApplicationException();
            }

            Application application =
                Application.create(
                    candidateId,
                    jobId,
                    ApplicationStatus.APPLIED
                );

            applicationRepository.save(application);

            statusHistoryRepository.save(
                ApplicationStatusHistory.initial(
                    application.getId(),
                    ApplicationStatus.APPLIED
                )
            );

            return mapper.toDto(application);
        }

Hands-on challenge

Implement duplicate prevention, status history and valid status transitions for job applications.

Quick check

Which statement is correct?

07 How do you handle external payment or banking failure?

Solution and explanation

Solution:

Suppose a university student has:

Semester Fee = Rs 80,000

The student pays Rs 30,000.

The database records:

invoice:
gross = 80000
paid = 30000
balance = 50000
status = PARTIALLY_PAID

Now the payment gateway returns:

SUCCESS

but the browser closes before the frontend receives the response.

The system must not assume:

"Browser closed = payment failed."

Instead, the backend should use the persisted payment intent and gateway reference.

Payment lifecycle:

INITIATED
|
PENDING
|
SUCCESS

or:

INITIATED
|
PENDING
|
CONFIRMED_FAILED

A webhook/callback should be authenticated and verified.

Duplicate callbacks should be idempotent.

If the same gateway reference arrives three times, the student should receive only one financial credit.

If the gateway confirms Rs 30,000 but the local database update fails, a database rollback cannot undo the gateway transaction.

The payment remains an external financial fact that must be reconciled.

Run this example

        @Transactional
        public void confirmPayment(
                String gatewayReference,
                BigDecimal amount
        ) {

            Payment payment =
                paymentRepository
                    .findByGatewayReferenceForUpdate(
                        gatewayReference
                    )
                    .orElseThrow();

            if (payment.isSuccessful()) {
                return;
            }

            if (payment.getAmount()
                .compareTo(amount) != 0) {

                throw new PaymentAmountMismatchException();
            }

            payment.markSuccessful();

            paymentRepository.save(payment);

            FeeInvoice invoice =
                invoiceRepository
                    .findByIdForUpdate(
                        payment.getInvoiceId()
                    )
                    .orElseThrow();

            invoice.addPayment(amount);

            invoiceRepository.save(invoice);
        }

Hands-on challenge

Implement an idempotent payment callback where the same gateway reference can be delivered multiple times.

Quick check

Which statement is correct?

08 What is retry and why can blindly retrying payments be dangerous?

Solution and explanation

Solution:

Retries are useful for transient failures.

Example:

Database connection temporarily fails.

Retry may succeed.

But financial operations are different.

Suppose:

POST /bank/transfer

Rs 50,000

The request reaches the bank.

The bank successfully transfers the money.

Before the response reaches our application, the network times out.

Our application sees:

TIMEOUT

If it blindly retries:

POST /bank/transfer Rs 50,000

the bank might transfer another Rs 50,000.

The customer could lose Rs 100,000 instead of Rs 50,000.

Therefore payment requests need idempotency.

Example:

businessReference = TRANSFER-2026-00001

The same reference must represent the same business transaction across retries.

If the bank supports querying status:

Query original reference.
Confirm outcome.
Retry only when safe.

Never convert UNKNOWN into FAILED merely because a network timeout occurred.

Run this example

        public TransferResult transfer(
                String businessReference,
                BigDecimal amount
        ) {

            Transfer existing =
                transferRepository
                    .findByReference(businessReference)
                    .orElse(null);

            if (existing != null) {
                return existing.toResult();
            }

            Transfer transfer =
                transferRepository.createPending(
                    businessReference,
                    amount
                );

            bankClient.submit(
                businessReference,
                amount
            );

            return transfer.toResult();
        }

Hands-on challenge

Simulate a successful bank transaction followed by a network timeout and verify that the retry does not create a second transfer.

Quick check

Which statement is correct?

09 How do you design a complete production Spring Boot architecture?

Solution and explanation

Solution:

Consider the complete career platform.

Users:

Job seekers Recruiters HR Administrators

Core services:

Identity Service Company Service Job Service Application Service Interview Service Notification Service Payment Service Learning Service Analytics Service

Request flow:

React/Angular

v Load Balancer

v API Gateway

+---- Security

+---- Job Service

+---- Application Service

+---- Company Service

+---- Learning Service

Application events:

Application Service

v Kafka

+---- Notification +---- Analytics +---- Audit +---- Recommendation

Storage:

PostgreSQL/MySQL

+---- authoritative business data

Redis

+---- cache +---- rate limits +---- temporary state

OpenSearch/Elasticsearch

+---- job search

Observability:

Spring Boot Actuator

Prometheus

Grafana

Deployment:

Docker

Kubernetes

Important production principles:

Each service owns its business data.
Database transactions protect local consistency.
Kafka handles asynchronous events.
Idempotency protects repeated requests.
Redis accelerates repeated reads.
Search engine handles complex search.
Distributed tracing follows requests.
Metrics identify bottlenecks.
Retries are bounded.
Financial operations are reconciled.
Sensitive APIs are authenticated and authorized.
Failures are represented explicitly instead of silently ignored.

Run this example

        Client
           |
           v
        Load Balancer
           |
           v
        API Gateway
           |
           +-------------------+
           |                   |
        Security            Routing
           |                   |
           +---------+---------+
                     |
           +---------+----------+----------+
           |         |          |          |
        Job       Company   Application  Learning
        Service   Service    Service     Service
           |         |          |
           |         |          +------+
           |         |                 |
           +---------+-------------- Kafka
                     |                 |
                 PostgreSQL      +-----+------+
                     |           |     |      |
                   Redis      Email Audit Analytics
                     |
              Search Index

        Observability:
        Actuator -> Prometheus -> Grafana

        Deployment:
        Docker -> Kubernetes

Hands-on challenge

Design the complete architecture for the career portal and identify the failure boundary of every major component.

Quick check

Which statement is correct?