Tokens in Java Explained: Types, Functions & How They Work

In Java, tokens are the smallest individual units of a program that hold meaning to the compiler. Just like words and punctuation marks form sentences in a language, tokens define the structure and syntax of Java code They are sequences of characters grouped together to create meaningful instructions that the Java compiler can analyse and execute.

Example of Java Token

int number = 10;
This line is made up of multiple tokens, these are:

• int – A keyword that tells the compiler the data type of the variable.
• number – An identifier, which is the name of the variable.
• = – An operator that assigns a value to the variable.
• 10 – A literal, representing a fixed value.
• ; – A separator that indicates the end of the statement.

Each of these tokens in java has a specific role and helps the compiler understand what the programmer is trying to perform. Without tokens, the program would be a mix of characters with no structure or meaning.

Java tokens are classified into five main types, each performing a specific purpose in a program's structure and functionality. Understanding these types is important for writing clear and error-free code. These types are:

• Keywords
• Identifiers
• Literals
• Operators
• Separators

Each token has a specific function that helps the compiler understand the structure and operations within the code. Here's a how we can explain java tokens:

1. Keywords in Java

Keywords in Java are reserved words with predefined meanings or more simply put, these are words defined in Java that represent a set of instructions. 

Features of Keywords 

• The keywords represent groups of instructions for the compiler. 
• These words are added to the structure of Java programming and cannot be used for naming variables, methods, or other identifiers. 
• These are always written in the lowercase.

Keywords help the compiler understand the operations and flow of the program. They define the syntax and indicate the type of behavior or action that needs to be performed.

‍Examples of Keywords:

• public, class, static, void: Define the structure of a Java program.
• if, else, switch: Control the flow of the program.
• int, double, boolean: Specify data types for variables.
• return: Used to send a value back from a method.

Example Code using keywords:

public class Example {
    public static void main(String[] args) {
        if (true) {
            System.out.println("This is a keyword example.");
        }
    }
}

In this code, public, class, static, void, and if are keywords. They define the program's structure and logic.

2. Names or Identifiers in Java

Identifiers in Java are the names used to identify variables, methods, classes, or objects. They are important for creating readable, maintainable, and functional programs, as they act as labels that the compiler and developers can use to reference specific elements within the code. 

For example, in the declaration int age = 25; the word age is an identifier representing the variable storing the value 25.

Rules for Naming Identifiers:

Java implements specific rules to confirm identifiers are valid and work correctly:

Starting Character: Identifiers must start with a letter (A-Z or a-z), an underscore (_), or a dollar sign ($).

Valid: myVariable, _tempValue, $amount
Invalid: 1variable, @name

1. Allowed Characters: After the first character, identifiers can include letters, digits (0-9), underscores, and dollar signs. No spaces or special characters (like #, @, &) are allowed.

2. It should not be Boolean literal, that is, true or false.

3. It should not be null literal.

4. No Keywords: Identifiers cannot use reserved Java keywords, such as class, int, or return.

5. No Length Restrictions: Technically, there is no limit to the length of an identifier, but it is recommended to keep them brief and defining for readability.

Example Code using Identifiers:

public class IdentifierExample {
    public static void main(String[] args) {
        int age = 25;  // 'age' is an identifier for a variable
        String name = "John";  // 'name' is another identifier
        System.out.println("Age: " + age);
        System.out.println("Name: " + name);
    }
}

The word ‘age' is an identifier for a variable, and 'name' is another identifier in the above example. 

3. Literals in Java

Literals in Java are fixed values written directly in the code, representing constants that do not change during program execution.
These values are used to assign specific data to variables or define constants in the program. For example, in the declaration int num = 100; the number 100 is a literal representing an integer value. The value may be a number, whether a whole number or decimal or it could also be a sequence of characters, which is called String literal, Boolean type, etc.

Literals are fundamental in programming as they act as the primary building blocks of data manipulation. Java provides several types of literals that provide different kinds of data requirements, such as numbers, text, and logical values.

Example Code:

public class LiteralExample {
    public static void main(String[] args) {
        int num = 100;               // Integer literal
        double pi = 3.14159;         // Floating-point literal
        char grade = 'A';            // Character literal
        String greeting = "Hello";  // String literal
        boolean isPassed = true;     // Boolean literal
        
        System.out.println("Number: " + num);
        System.out.println("Pi: " + pi);
        System.out.println("Grade: " + grade);
        System.out.println("Greeting: " + greeting);
        System.out.println("Passed: " + isPassed);
    }
}

Output:

Number: 100  
Pi: 3.14159  
Grade: A  
Greeting: Hello  
Passed: true  

Output Explained:

The code demonstrates the use of different types of literals in Java. The output prints the values of an integer literal (100), a floating-point literal (3.14159), a character literal (A), a string literal (Hello), and a boolean literal (true). Each line corresponds to the respective literal and displays it as part of a message.

Complexity of the Code:

The time complexity of this code is O(1), as it only performs a constant number of operations: variable initialization and printing outputs.

Types of Literals in Java

There are five types of Literals on Java:

1. Integer Literals

Integer Literals in Java refer to fixed values used in the code, representing whole numbers. They can be written in various number systems such as decimal, hexadecimal, octal, and binary.

Decimal Integer Literals are sequences of decimal digits (0-9) and are the most common way to represent integers. Examples include values like 6, 453, or 34789.

Hexadecimal Integer Literals use a sequence of hexadecimal digits (0-9, A-F) to represent values. These numbers are prefixed with 0x or 0X to indicate that they are in base 16. Examples include 0x56ab, 0X6AF2, etc.

Octal Integer Literals are sequences of digits from 0 to 7 and are prefixed with a 0 to denote base 8. Examples of octal literals include 07122, 04, and 043526.

Binary Integer Literals consist of binary digits (0 and 1) and are prefixed with 0b or 0B to indicate base 2. Examples include 0b0111001, 0b101, and 0b1000.

2. Floating-Point Literals

Floating-point literals represent numbers with decimal points or in exponential notation. These are used for specific values, such as measurements or calculations with fractions. Float literals are suffixed with f or F, while double literals use d or D. Examples include 3.14, -0.5, and 2.5e3 (scientific notation).

3. Character Literals

Character literals represent a single character in single quotes ('). They can include letters, digits, symbols, or escape sequences like '\n' (newline) and '\t' (tab). These literals are useful for handling individual characters in text processing. Examples include 'A', 'z', and '#'.

4. String Literals

String literals represent sequences of characters in double quotes ("). They are used for handling text data, such as messages, user input, or file content. Examples include "Hello", "Java Programming", and "12345". 

5. Boolean Literals

Boolean literals represent logical values, either true or false. They are used in conditional statements and logical operations to control program flow. For example, if (isPassed), the variable isPassed could be assigned a Boolean literal, such as true. Examples are true and false.

6. Null Literal

There is only one value of Null Literal, that is, null.

4. Operators in Java

Operators in Java are symbols used to perform operations on variables, values, or expressions. They allow developers to manipulate data, make comparisons, and assign values to variables. 

Java offers many types of operators which classified based on the type of operation they perform.

Operators are important in almost every program as they enable the manipulation of data, control the flow of execution, and help in assessing conditions.

Example Code:

public class OperatorExample {
    public static void main(String[] args) {
        int a = 10;
        int b = 20;
        
        int sum = a + b;  // Addition operator
        boolean isGreater = a > b;  // Comparison operator
        boolean isEqual = a == b;  // Comparison operator
        
        System.out.println("Sum: " + sum);  // 30
        System.out.println("Is a greater than b? " + isGreater);  // false
        System.out.println("Is a equal to b? " + isEqual);  // false
    }
}

Output:

Sum: 30  
Is a greater than b? false  
Is a equal to b? false  

Types of Operators in Java

There are four types of operators in java:

1. Arithmetic Operators

Arithmetic operators are used to perform basic mathematical operations like addition, subtraction, multiplication, division, and modulus (remainder).

Example:

int a = 10;
int b = 20;
int sum = a + b;  // sum will be 30
int remainder = b % a;  // remainder will be 0

2. Comparison or Relational Operators 

Comparison operators are used to compare two values and return a boolean result (true or false). These operators help assess conditions and make decisions in a program.

Example:

int a = 10;
int b = 20;
boolean isEqual = a == b;  // false
boolean isGreater = a > b;  // false
boolean isLessThanOrEqual = a <= b;  // true

3. Logical Operators

Logical operators are used to combine multiple boolean expressions and return a single boolean value. They help in making complex decisions and controlling program flow.

• && (Logical AND): Returns true if both operands are true.
• || (Logical OR): Returns true if at least one operand is true.
• ! (Logical NOT): Inverts the boolean value of the operand.

Example:

boolean isAdult = true;
boolean hasLicense = false;
boolean canDrive = isAdult && hasLicense;  // false
boolean isNotAdult = !isAdult;  // false

4. Assignment Operators

Assignment operators are used to assign values to variables. In addition to simple assignment, Java provides compound assignment operators that combine assignment with an arithmetic operation.

Example:

int a = 10;
a += 5;  // a becomes 15 (a = a + 5)
a -= 3;  // a becomes 12 (a = a - 3)
a *= 2;  // a becomes 24 (a = a * 2)
a /= 4;  // a becomes 6 (a = a / 4)
a %= 3;  // a becomes 0 (a = a % 3)

5. Increment / Decrement Operators:

6. Bitwise Operators:

7. Conditional Operators:

The ? operator is used to construct a conditional expression, also known as the ternary operator, that evaluates a condition and returns one of two values based on the result.

Key Points About Operators

1. Precedence: Operators have a specific precedence order, which indicates the sequence in which operations are performed. For example, multiplication and division have higher precedence than addition and subtraction.
2. Associativity: Most operators are left-associative, meaning they are considered from left to right. However, assignment (=) and conditional operators (? :) are right-associative.
3. Overloading: Java supports operator overloading for specific types which allow custom behavior in specific classes, but not for arithmetic operators themselves.

5. Separators in Java

Separators in Java are symbols that help define the structure of your code and organize different parts of a program. These symbols denote boundaries, group elements together, and separate different components within the program. Without separators, the program would lack clarity and structure, making it hard for both the compiler and developers to understand the flow of the code.Java provides several types of separators that each serve a specific purpose, helping break down code into manageable parts, control execution flow, and manage collections of data.

Example Code:

public class SeparatorExample {
    public static void main(String[] args) {
        int[] numbers = {1, 2, 3}; // {} for array initialization and [] for array declaration
        System.out.println(numbers[0]); // [] for array access
    }
}

Common Separators in Java

6. Comments 

In Java, comments are used to provide explanatory notes or descriptions within the code. These comments are ignored by the compiler and do not affect the execution of the program. 

There are two types of comments in Java:

Single-line comments (//): 

Single line comments are used for brief explanations, typically for one line of code.

Example: // This is a single-line comment

Multi-line comments (/* ... */): 

 Multi-line comments are used for longer explanations spanning multiple lines.

The process of compiling Java code involves breaking down the source code into these tokens, analyzing them, and confirming that they form a valid and executable program. Here's how Java tokens work:

1. Java Tokenization

Java tokenization is the first step in the compilation process. During this phase, the compiler reads the source code character by character and breaks it into individual tokens. Tokenize java string can represent various elements such as keywords, identifiers, literals, operators, and separators. 

For Example:

int x = 10;

The compiler identifies int, x, =, 10, and ; as separate tokens.

2. Syntax Analysis

Once the code is tokenized, the next step is syntax analysis. In this phase, the compiler checks if the sequence of tokens follows the rules of Java syntax. Syntax rules define how tokens can be combined to form valid statements or expressions. If the tokens are arranged incorrectly, the compiler will generate a syntax error.

3. Semantic Analysis

After syntax analysis, the compiler performs semantic analysis. This step confirms that the tokens make logical sense in the context of the program. The compiler checks that the program’s structure is valid in terms of variable declarations, data types, and operations. 

For Example:

int x = "hello";

Here, the semantic analysis will detect an error because:

• The variable x is declared as an int (integer type), but "hello" is a string literal, which is incompatible with the int type.
• The compiler will raise a semantic error, such as "incompatible types: String cannot be converted to int."

Here’s a program to see how different types of tokens come together:

public class TokenDemo {
    public static void main(String[] args) {
        // Declaring variables
        int num1 = 15;
        int num2 = 20;
        
        // Using an operator
        int sum = num1 + num2;
        
        // Printing the result
        System.out.println("The sum is: " + sum);
    }
}

Tokens in the Example:

• Keywords: public, class, int, static, void
• Identifiers: TokenDemo, main, num1, num2, sum
• Literals: 15, 20, "The sum is: "
• Operators: =, +
• Separators: {, }, ;, (, )

A JSON Web Token (JWT) is a compact, self-contained way to securely transmit information between two parties, such as a client and a server. It is a string composed of three parts—header, payload, and signature—separated by dots. 

The header contains metadata about the token, like the algorithm used for signing, while the payload holds the actual information (claims) being transmitted. The signature ensures that the token hasn’t been altered, providing authenticity and integrity.

A Bearer Token is a type of token used in authentication and authorization mechanisms. It is a part of the HTTP header used in API requests, commonly for passing authentication information like JWT (JSON Web Tokens). The bearer token is included in the Authorization header of HTTP requests to authenticate users or applications.

1. Add Dependencies (Maven): 

If you're working with JWT in Java, you'll need libraries to generate and verify the JWTs. The commonly used library is jjwt. Add the dependency in your pom.xml file.

<dependency>
    <groupId>io.jsonwebtoken</groupId>
    <artifactId>jjwt</artifactId>
    <version>0.11.5</version>
</dependency>

2. Generating a JWT Token (Bearer Token): 

To generate a bearer token, you'll need to create a JWT. Here’s how you can create java token generator, a simple JWT token using RS256 (RSA encryption) in Java. 

JWT Java example:

import io.jsonwebtoken.Jwts;
import io.jsonwebtoken.SignatureAlgorithm;

import java.util.Date;

public class JWTGenerator {

    private static final String SECRET_KEY = "your-256-bit-secret"; // Ideally, store this securely

    public static String generateToken(String username) {
        return Jwts.builder()
                .setSubject(username)
                .setIssuedAt(new Date())
                .setExpiration(new Date(System.currentTimeMillis() + 3600000))  // 1 hour expiration
                .signWith(SignatureAlgorithm.HS256, SECRET_KEY)
                .compact();
    }

    public static void main(String[] args) {
        String token = generateToken("user123");
        System.out.println("Bearer Token: " + token);
    }
}

HS256 is used for signing the JWT. You can replace it with RS256 if you're using RSA encryption.

Expiration: The token expires in 1 hour.

3. Sending Bearer Token in HTTP Request: 

Once you have the token, it’s time to send it in an HTTP request header. Here is a sample of bearer token java code using HttpURLConnection to send a GET request with the bearer token.

import java.net.HttpURLConnection;
import java.net.URL;

public class BearerTokenRequest {

    public static void sendRequest(String token) throws Exception {
        // Create URL and open connection
        URL url = new URL("https://your-api.com/protected-endpoint");
        HttpURLConnection connection = (HttpURLConnection) url.openConnection();
        connection.setRequestMethod("GET");

        // Add Authorization header with Bearer token
        connection.setRequestProperty("Authorization", "Bearer " + token);

        // Get response
        int responseCode = connection.getResponseCode();
        System.out.println("Response Code: " + responseCode);
    }

    public static void main(String[] args) throws Exception {
        String token = "your-jwt-token"; // Use the generated token
        sendRequest(token);
    }
}

4. Verify the Token on Server Side: 

On the server side, you’ll need to verify the token. Here’s an authorization bearer token java example of how to verify a JWT token using jjwt java:

import io.jsonwebtoken.Jwts;
import io.jsonwebtoken.Claims;
import io.jsonwebtoken.SignatureException;

public class JWTVerifier {

    private static final String SECRET_KEY = "your-256-bit-secret";

    public static Claims verifyToken(String token) throws SignatureException {
        return Jwts.parser()
                .setSigningKey(SECRET_KEY)
                .parseClaimsJws(token)
                .getBody();
    }

    public static void main(String[] args) {
        String token = "your-jwt-token";
        try {
            Claims claims = verifyToken(token);
            System.out.println("Token is valid. User: " + claims.getSubject());
        } catch (SignatureException e) {
            System.out.println("Invalid token.");
        }
    }
}

This code checks the signature of the token using the secret key. If the token is valid, the server can extract claims (such as the subject or username) and proceed.

Java tokens form the foundational building blocks of any Java program which enable both the compiler and the developer to understand the structure and functionality of the code. These tokens, which include keywords, identifiers, literals, operators, and separators, help organize and define the instructions that a program executes. Keywords set the structure and flow, identifiers give meaning to variables and functions, literals provide constant values, operators perform necessary operations, and separators break down the code into comprehensible units.

1. What is a type token in Java?

A type token in Java refers to the classification of a token that represents a specific data type, such as an integer, floating-point number, or boolean. It helps the compiler understand what kind of value a variable can hold.

2. What are the punctuators and separators in Java?

In Java, punctuators and separators are symbols that help structure and organize the code. Punctuators include operators and symbols like +, -, *, =, while separators are symbols that help define boundaries or groupings, such as parentheses (), curly braces {}, semicolons ;, and commas ,.

3. What is a token in Java string?

In a Java string, a token refers to a sequence of characters that can be parsed as a meaningful unit. These tokens can be words, symbols, or other structures that the Java compiler identifies within the string for processing, such as keywords, identifiers, and literals.

4. Why are tokens important in Java?

Tokens are important because they help the Java compiler break down the source code into manageable parts for analysis and execution. Understanding tokens is important for writing syntactically correct Java code and for debugging purposes.

5. How does the Java compiler identify tokens?

The Java compiler identifies tokens through a process called tokenization or lexical analysis. During this process, the compiler scans the source code, breaks it into individual tokens, assigns meanings to each token, and prepares them for further analysis and compilation.

6. Are tokens case-sensitive in Java?

Yes, tokens in Java are case-sensitive. This means that uppercase and lowercase letters are treated as distinct characters. For example, Variable and variable would be considered two different identifiers.

7. Can I create my own tokens in Java?

In Java, you cannot create custom tokens as they are predefined by the language specification. However, you can define your own identifiers (such as variable names and method names) within the constraints of the language rules.