One Crore Data Processing System

Section 1

Ground Zero Introduction

Start here

What problem are we trying to solve?

Suppose we have a very large collection of integers. It might contain one crore numbers, which means ten million values. If the data is very large, reading and sorting everything together may become inconvenient. So this project explores file-based processing.

Simple meaning of file-based processing

Instead of asking the computer to hold everything at once, we divide the work by files or chunks, process one piece at a time, and combine results later.

Two designs inside this project

The active code computes local and global min-max across files. The backup code goes further and recursively breaks a giant file into smaller range-based files.

Section 2

Big Picture of the Whole System

Architecture first

Huge Numeric Data

→

Split or Store in Files

→

Process File by File

→

Combine Results

Local Min-Max

Range Partitioning

Recursive Splitting

Final Output

There are really two levels of thinking here:

Level 1:
Many files already exist
→ scan each file
→ find each file's min and max
→ combine results

Level 2:
One giant file exists
→ split by numeric midpoint
→ create child files
→ recurse until each child file is small enough

Section 3

The Active Program

What currently runs

Main truth about the active code

The current main program is not a full sorter. It reads many files, finds the min and max in each file, and then computes the overall min and max across all files.

Data folder
  ↓
Get all files
  ↓
For each file:
    open file
    read integers
    compute local min-max
    store FileData(file, mm)
  ↓
Compare all file summaries
  ↓
Print overall max and min

Section 3.1

Getting All Files from the Data Folder

Relevant real code

Meaning

This method takes a folder path, creates a Java File object for that directory, and returns the files inside it.

Actual code from BillionsAndBillions.java

public static File[] getAllFiles(String directory)
{
    try
    {
        File f=new File(directory);
        return f.listFiles();
    }
    catch(Exception ex)
    {
        System.out.println(ex);
        return null;
    }
}

Section 3.2

Scanning One File and Finding Its Min and Max

Core method

1

Open the file using `Scanner`

The file is read integer by integer.

2

Use the first number to initialize min and max

That is why the boolean b exists.

3

Compare every later number

Update max if larger; update min if smaller.

4

Return a `FileData` object

This stores the file and its local result.

Actual code from BillionsAndBillions.java

public static FileData getFileData(File file)
{
    try
    {
        MaxMin mm=new MaxMin();
        boolean b=false;
        Scanner s=new Scanner(file);
        while(s.hasNextInt())
        {
            int n=s.nextInt();
            if(!b)
            {
                mm.max=n;
                mm.min=n;
            }
            else
            {
                if(n>mm.max)
                    mm.max=n;
                if(n

        
          
            Initialization part
            if(!b)
{
    mm.max=n;
    mm.min=n;
}
            
              The first number is used to set both min and max.
            
          

          
            Comparison part
            else
{
    if(n>mm.max)
        mm.max=n;
    if(n
            
              Every later number is checked against the current min and max.
            
          
        

        
          Dry run idea
          
            If 0.txt contains values from 0 to 9, then the first value
            sets both min and max to 0, and after scanning all values, max becomes 9
            while min stays 0.

Section 3.3

Building the Array of File Summaries

Collect local results

Meaning

This method first gets all files, then scans each one, and stores the result in an array of FileData objects.

Actual code from BillionsAndBillions.java

public static FileData[] getFilesData(String directory)
{
    File[] files=getAllFiles(directory);
    FileData[] filedata=new FileData[files.length];
    for(int i=0;i<=files.length-1;i++)
    {
        File f=files[i];
        FileData fd=getFileData(f);
        filedata[i]=fd;
    }
    return filedata;
}

[ (0.txt, Max=9, Min=0), (1.txt, Max=19, Min=10), (2.txt, Max=29, Min=20), ... (9.txt, Max=99, Min=90) ]

Section 3.4

Computing the Overall Min and Max

Global combine step

Meaning

This method compares all the file-level summaries. It is correct, but it uses nested loops, which means it does more repeated work than necessary.

Actual code from BillionsAndBillions.java

public static MaxMin getTotalMaxMin(FileData[] fd)
{
    boolean b=false;
    int max=0;
    int min=0;
    for(int i=0;i<=fd.length-1;i++)
    {
        max=fd[i].mm.max;
        min=fd[i].mm.min;
        for(int j=0;j<=fd.length-1;j++)
        {
            if(fd[j].mm.max>max)
                max=fd[j].mm.max;
            if(fd[j].mm.min

        
          
            Comparison logic
            if(fd[j].mm.max>max)
    max=fd[j].mm.max;
if(fd[j].mm.min
          

          
            Returned final object
            MaxMin mm=new MaxMin();
mm.max=max;
mm.min=min;
return mm;
          
        

        
          Result for the sample chunk files
          
            If the files are 0.txt through 9.txt with ranges 0–9,
            10–19, and so on up to 90–99, then the final result is:
          
          Max=99 ,Min=0

Section 3.5

The Active Program Entry Point: `main()`

How execution starts

Actual code from BillionsAndBillions.java

public static void main(String[] args)
{
    FileData[] data=getFilesData("Data");
    for(FileData datum : data)
        System.out.println(datum);

    MaxMin totalmaxmin=getTotalMaxMin(data);
    System.out.println(totalmaxmin);
}

Section 4

Supporting Classes

Data holders

Actual code from FileData.java

public class FileData
{
    public java.io.File file;
    public MaxMin mm;

    public FileData(java.io.File file,MaxMin mm)
    {
        this.file=file;
        this.mm=mm;
    }

    public String toString()
    {
        return "" + file + " , " + mm;
    }
}

This stores the file and its corresponding MaxMin result.

Actual code from MaxMin.java

public class MaxMin
{
    public int max,min;

    public String toString()
    {
        return "Max=" + max + " ,Min=" + min;
    }
}

This stores only two values: max and min.

Section 5

The Backup Recursive System

The bigger design idea

Main idea

The backup code is trying to solve a larger problem. Instead of just finding min and max in many files, it starts with one very large file and recursively breaks it into smaller files according to numeric range.

Generate huge file
  ↓
Read min, max, size from filename
  ↓
If size is already small enough → stop
  ↓
Else compute midpoint
  ↓
Split values into two child files
  ↓
Update child min, max, and size
  ↓
Delete parent file
  ↓
Rename child files with metadata
  ↓
Recurse on both child files

Section 5.1

Generating the Huge Random Data File

Where the big file comes from

Actual code from OneCroreBreakUp.java

public static int getRandom(int lowerlimit,int upperlimit)
{
    if(lowerlimit==upperlimit)
        return lowerlimit;
    if(lowerlimit>upperlimit)
    {
        int t=lowerlimit;
        lowerlimit=upperlimit;
        upperlimit=t;
    }
    int range=upperlimit-lowerlimit + 1;
    int n=(int)(Math.random()*range);
    return lowerlimit + n;
}

Actual file-writing method

public static void saveArray(String onlyfilename,int n,int min,int max) 
{
    try
    {
        String filename=getFileName(onlyfilename + "0",min,max,n);
        PrintWriter pw=new PrintWriter("Data\\\\" + filename);
        for(int i=1;i<=n;i++)
        {
            String str="" + getRandom(min,max);
            pw.println(str);
        }
        pw.flush();
        pw.close();
    }
    catch(Exception ex)
    {
        System.out.println(ex);
    }
}

Section 5.2

Metadata Stored Inside the File Name

Important design trick

Actual code that creates the metadata-rich file name

public static String getFileName(String filename,int min,int max,int size)
{
    return filename + "," + min + "," + max + "," + size;
}

Example result: Data0,-1000,1000,10000000

Actual code that reads metadata back

public static FileData getDataFromFileName(String fullfilename)
{
    String[] strings=fullfilename.split(",");
    String filename=strings[0];
    int min=Integer.parseInt(strings[1]);
    int max=Integer.parseInt(strings[2]);
    int size=Integer.parseInt(strings[3]);
    return new FileData(min,max,size,filename,fullfilename);
}

Actual backup FileData.java

public class FileData
{
    public int min,max,size;
    public String filename,fullfilename;

    public FileData(int min,int max,int size,String filename,String fullfilename)
    {
        this.min=min;
        this.max=max;
        this.size=size;
        this.filename=filename;
        this.fullfilename=fullfilename;
    }

    public String toString()
    {
        return "Min = " + min + ", Max = " + max + ", Size = " + size + ", FileName = " + filename + ", FullFileName = " + fullfilename;
    }
}

Section 5.3

The Heart of the Backup System: `breakTheFile()`

Recursive splitting logic

What this method does

It checks if a file is already small enough. If not, it computes the midpoint of the numeric range, sends lower values into one child file and greater-or-equal values into another child file, updates metadata while writing, deletes the old file, and then recurses on the two new files.

Actual code from OneCroreBreakUp.java

public static void breakTheFile(String filename,int maxsize)
{
    try
    {
        FileData fd=getDataFromFileName(filename);
        if(fd.sizemax1)
                        max1=x;
                    if(xmax2)
                        max2=x;
                    if(x0)
        {
            pw1.flush();
            pw1.close();
        }

        if(n2>0)
        {
            pw2.flush();
            pw2.close();
        }

        s.close();
        currentfile.delete();

        String filename1=getFileName("" + fileno++,min1,max1,n1);
        String filename2=getFileName("" + fileno++,min2,max2,n2);

        totalfiles++;
        f1.renameTo(new File("Data\\\\" + filename1));
        System.out.println("File " + totalfiles);

        totalfiles++;
        f2.renameTo(new File("Data\\\\" + filename2));
        System.out.println("File " + totalfiles);

        if(n1>0)
            breakTheFile(filename1,maxsize);
        if(n2>0)
            breakTheFile(filename2,maxsize);
    }
    catch(Exception ex)
    {
        System.out.println(ex);
        System.exit(0);
    }
}

Stopping rule and midpoint

FileData fd=getDataFromFileName(filename);
if(fd.size

The actual split condition

if(x

Simple dry run example

If the file range is -1000 to 1000, then avg = 0. So values like -700 and -2 go left. Values like 0, 15, and 999 go right.

Section 5.4

The Backup Program Entry Point

How recursion starts

Actual code from OneCroreBreakUp.java

public static void main(String[] args)
{
    saveArray("Data",10000000,-1000,1000);
    breakTheFile("Data0,-1000,1000,10000000",25000);
}

First line meaning

Create a file containing ten million random integers between -1000 and 1000.

Second line meaning

Keep breaking that file until every resulting file has fewer than 25,000 numbers.

Section 6

Counting Sort Demo in the Backup Folder

Separate sorting example

Why this file matters

This is not the active large-file engine, but it demonstrates counting sort on a normal in-memory array.

Actual code from Counting.java

class Counting
{
    public static void main(String[ ] args)
    {
        int max=0;
        int b[ ];
        int[] a={21,3,54,10,54,67,89,21};
        for(int i=0;i<=a.length-1;i++)
        {
            if(max=0;i--)
        {
            int value=a[i];
            int pos=f[value];
            f[value]--;
            pos--;
            b[pos]=value;
        }
        for(int i=0;i<=a.length-1;i++)
            a[i]=b[i];
    }
}

Section 7

GitHub Repository and Running Java on GitHub

Practical publishing workflow

Placeholder GitHub Repository Link

Replace this dummy link with your real repository later.

Open GitHub Repository

What GitHub is good for here

Store the Java source code, version the project, publish docs and lesson pages, and use GitHub Actions to compile and test the Java project automatically.

Important limitation

GitHub Pages is for static content such as HTML, CSS, and JavaScript. It is a good place for your lesson page, project explanation, screenshots, and download links, but not for hosting a live Java backend.

Simple recommended setup

Put your Java project code in the GitHub repository. Put this HTML lesson in the same repository or in a docs branch/folder. Use GitHub Pages for the lesson page, and GitHub Actions for Java build/test automation.

Example GitHub Actions workflow for a simple Java compile

name: Java Build

on:
  push:
  pull_request:

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Set up Java
        uses: actions/setup-java@v4
        with:
          distribution: temurin
          java-version: '17'

      - name: Compile Java files
        run: javac *.java

Dummy HTML block for a repository section

<section class="card">
  <h2>Project Repository</h2>
  <p>Replace this dummy link with your real GitHub repository.</p>
  <a
    href="https://github.com/your-username/your-repository"
    target="_blank"
    rel="noopener"
    class="btn btn-primary"
  >
    View Project on GitHub
  </a>
</section>

Section 8

Running Java on Google Colab

Notebook workflow

Can Java run on Colab?

Yes. A Colab notebook is a Linux-based environment, so you can compile and run Java using shell commands. The usual flow is: check Java → upload ZIP → unzip → compile → run.

1

Check Java version

!java -version

2

Upload your ZIP file

from google.colab import files
files.upload()

3

Unzip the project

!unzip "One Crore Sorting.zip"

4

Compile the Java files

!javac *.java

5

Run the active program

!java BillionsAndBillions

Hello World test in Colab

%%writefile HelloWorld.java
public class HelloWorld {
    public static void main(String[] args) {
        System.out.println("Hello from Java in Colab!");
    }
}

!javac HelloWorld.java
!java HelloWorld

Custom project notes

If your ZIP contains multiple folders with duplicate class names, compile one target folder at a time. Otherwise Java compilation may clash.

Colab-ready project workflow block

# 1. Check Java
!java -version

# 2. Upload the zip
from google.colab import files
files.upload()

# 3. Unzip project
!unzip "One Crore Sorting.zip"

# 4. Optional: see Java files
!find . -name "*.java"

# 5. Compile top-level Java files
!javac *.java

# 6. Run active main class
!java BillionsAndBillions

# 7. Optional backup runs
# !java OneCroreBreakUp
# !java Counting

Section 9

Complexity and Critical View

Evaluate it properly

Active program complexity

Scanning all integers across files: O(N) Current combine step: O(F²)

Here N is the total number of integers and F is the number of files.

Backup recursive intuition

Rough intuition: O(N log R)

Here R is the numeric range being split recursively.

Strengths

Good large-data thinking, file-wise processing, clean local summaries, and a smart recursive range-based design in the backup folder.

Limitations

The active code does not fully sort the whole dataset. The global combine method is more repetitive than needed, and the backup design would still need a final production-ready sort/merge strategy if full ordering is desired.

Section 10

Practice MCQs

Check understanding

Section 11

Final Summary

Last takeaway

One-line summary

This project shows how very large numeric data can be processed through files: the active code computes local and global min-max values, while the backup design recursively partitions a giant file into smaller range-based files.

Best project setup

Use GitHub for source control and GitHub Actions for Java build/test automation. Use GitHub Pages for this lesson. Use Colab for quick demos, teaching, and simple compile-run workflows.

Best next upgrade

Add a proper final sorting or merge stage so the backup design becomes a more complete external sorting system.

Ground Zero Introduction

What problem are we trying to solve?

Simple meaning of file-based processing

Two designs inside this project

Big Picture of the Whole System

The Active Program

Main truth about the active code

Getting All Files from the Data Folder

Meaning

Scanning One File and Finding Its Min and Max

Open the file using Scanner

Use the first number to initialize min and max

Compare every later number

Return a FileData object

Dry run idea

Building the Array of File Summaries

Meaning

Computing the Overall Min and Max

Meaning

Result for the sample chunk files

The Active Program Entry Point: main()

Supporting Classes

The Backup Recursive System

Main idea

Generating the Huge Random Data File

Metadata Stored Inside the File Name

The Heart of the Backup System: breakTheFile()

What this method does

Simple dry run example

The Backup Program Entry Point

First line meaning

Second line meaning

Counting Sort Demo in the Backup Folder

Why this file matters

GitHub Repository and Running Java on GitHub

Placeholder GitHub Repository Link

What GitHub is good for here

Important limitation

Simple recommended setup

Running Java on Google Colab

Can Java run on Colab?

Check Java version

Upload your ZIP file

Unzip the project

Compile the Java files

Run the active program

Hello World test in Colab

Custom project notes

Complexity and Critical View

Active program complexity

Backup recursive intuition

Strengths

Limitations

Practice MCQs

1. What does the active program mainly compute?

2. What is the role of boolean b in getFileData()?

3. In the backup system, how is the split point chosen?

4. What is GitHub Pages best used for in this project?

5. What is a practical Java workflow on Colab?

Final Summary

One-line summary

Best project setup

Best next upgrade

Open the file using `Scanner`

Return a `FileData` object

The Active Program Entry Point: `main()`

The Heart of the Backup System: `breakTheFile()`

2. What is the role of boolean `b` in `getFileData()`?