Major Project Report (3).pdf

ENHANCED Q-LEARNING DQMAC PROTOCOL FOR IOT NETWORKS A Thesis Submitted in partial fulfillment of the requirements for the award of the Degree of BACHELOR OF TECHNOLOGY IN ELECTRONICS & COMMUNICATIO...

ENHANCED Q-LEARNING DQMAC PROTOCOL FOR IOT NETWORKS A Thesis Submitted in partial fulfillment of the requirements for the award of the Degree of BACHELOR OF TECHNOLOGY IN ELECTRONICS & COMMUNICATION ENGINEERING BY ABHISHEK KUMAR (BTECH/15094/20) SHANTI KUMARI (BTECH/15120/20) DEPARTMENT OF ELECTRONICS & COMMUNICATION ENGINEERING BIRLA INSTITUTE OF TECHNOLOGY, MESRA, PATNA CAMPUS, BIHAR- 800014, INDIA i  APPROVAL OF THE GUIDE Recommended that the thesis entitled “Enhanced Q-Learning based DQMAC protocol for IOT networks” presented by Abhishek Kumar and Shanti Kumari under my supervision and guidance be accepted as fulfilling this part of the requirements for the award of Degree of Bachelor of Technology in Electronics & Communication Engineering. To the best of my knowledge, the content of this thesis did not form a basis for the award of any previous degree to anyone else. Date: ____________ __________________ Dr.Atul Kumar Pandey Assistant Professor Dept.of Electronics & Communication Engineering Birla Institute of Technology Mesra, Patna ii  DECLARATION CERTIFICATE I certify that a) The work contained in the thesis is original and has been done by myself under the general supervision of my supervisor. b) The work has not been submitted to any other Institute for any other degree or diploma. c) I have followed the guidelines provided by the Institute in writing the thesis. d) I have conformed to the norms and guidelines given in the Ethical Code of Conduct of the Institute. e) Whenever I have used materials (data, theoretical analysis, and text) from other sources, I have given due credit to them by citing them in the text of the thesis and giving their details in the references. f) Whenever I have quoted written materials from other sources, I have put them under quotation marks and given due credit to the sources by citing them and giving required details in the references. ABHISHEK KUMAR (BTECH/15094/20) SHANTI KUMARI (BTECH/15120/20) iii  CERTIFICATE OF APPROVAL This is to certify that the work embodied in this thesis entitled “Enhanced Q-Learning based DQMAC Protocol for IOT networks”, is carried out by Abhishek Kumar (BTECH/15094/20) and Shanti Kumari (BTECH/15120/20) has been approved for the degree of Bachelor of Technology in Electronics & Communication Engineering at Birla Institute of Technology, Mesra, Patna. Date: Place: Internal Examiner External Examiner Head of Department iv  ABSTRACT The Enhanced Q-Learning-based Distributed Queueing (EQL-DQ) algorithm represents a novel approach to Medium Access Control (MAC) protocol design tailored specifically for Internet-of-Things (IoT) networks. In this project, we present a comprehensive overview of the EQL-DQ algorithm, detailing its core principles, design considerations, and performance evaluation. The project begins by introducing the motivation behind the development of EQL-DQ, highlighting the growing importance of efficient MAC protocols in IoT environments characterized by diverse traffic types, dynamic network conditions, and stringent energy constraints. We outline the key challenges faced by existing MAC protocols and articulate the need for a more adaptive, high throughput, low latency and energy-efficient approach. Next, we delve into the core components of the EQL-DQ algorithm, starting with an explanation of its reinforcement learning-based adaptive adjustment mechanism for the number of contending nodes. We describe how EQL-DQ dynamically optimizes contention parameters to balance throughput, delay, and energy consumption, ensuring efficient resource utilization and improved network performance. Furthermore, we elucidate the distributed queueing mechanism employed by EQL-DQ to handle collisions and parallel transmissions of data. By leveraging distributed queues, EQL- DQ optimizes the allocation of transmission opportunities, minimizing contention and enhancing overall system throughput and fairness. The project also highlights the parallel execution of contention slots as a key feature of EQL- DQ, enabling efficient utilization of computational resources and enhancing simulation performance. We discuss the rationale behind parallel execution and its implications for scalability and simulation accuracy in evaluations. To empirically evaluate the performance of EQL-DQ, extensive simulations were conducted, comparing its performance against existing Q-learning-based MAC protocols and traditional approaches. Throughput, average contentions, average delay, and energy consumption metrics were analyzed to demonstrate the superiority of EQL-DQ in various network scenarios. v  ACKNOWLEDGEMENT An endeavour over a long period can be successful only with the advice and support of many well-wishers. The task would be incomplete without mentioning the people who have made it possible, because is the epitome of hard work. So, with gratitude, we acknowledge all those whose guidance and encouragement owned our efforts with success. We confirm that this project report is the original and independent work of Abhishek Kumar and Shanti Kumari. It has not been submitted in part or whole for any other degree or academic purpose. All the information and data presented in this report are genuine and have been obtained from reliable, duly acknowledged sources. We wish to express grateful acknowledgment to our guide Dr. Atul Kumar Pandey, B.Tech., M.E., Ph.D., Assistant Professor, Department of Electronics & Communications Engineering, BIT PATNA for his inspiring guidance and continuous encouragement throughout the project. Finally, we would like to extend our deep sense of gratitude to all the friends, and last but not least we are greatly indebted to our parents who inspired us under at all circumstances. Thank you. DATE: Abhishek Kumar (BTECH/15094/20) Shanti Kumari (BTECH/15120/20) vi  CONTENTS ABSTRACT v ACKNOWLEDGMENT vi LIST OF FIGURES ix LIST OF TABLES x 1 INTRODUCTION …………………………………………………………………...1 1.1. INTRODUCTION TO IOT NETWORK PROTOCLS………………………………………..1 1.2. INTRDOCTION OF MAC PROTOCOL………………………………………………….…3 1.3. CLASSIFICATION OF WSN MAC PROTOCOL………………………………………….…….5 1.4. CHARACTERISTICS OF MAC PROTOCOLS …..…….…………………………...…………10 1.5. GOALS OF MAC PROTOCOLS ……….……………………...……………..……………….11 1.6. IEEE 802.15.4 MAC …………………….………..…………………………………………...11 1.6.1 APPLICATIONS OF IEEE 802.15.4………………………………………………....…...11 1.6.2 SUPERFRAME STRUCTURE..........................................................................................12 1.6.3 GTS MANAGEMENT ……………………………………………………………........13 1.6.4 DATA TRANSFER ………………………………………………………….................14 1..6.5 SLOTTED CSMA-CA PROTOCOL………………………………………………………..15 1.7 DQ MECHANISM ………………….………………………………………………….………15 1.8 MACHINE LEARNING ………………………………………………………………..............17 1.8.1 THE THREE MAIN TYPES OF MACHINE LEARNING ALGORITHM AND THEIR , APPLICATIONS IN COMMUNICATION NETWORK…………………………………………18 1.9 INTRODUCTION TO Q-LEARNING ……………………...……………………….…………20 1.10 INTRODUCTION EQL-DQ………………………………………………………………………22 2 LITERATURE REVIEW……………………………………………………………23 3 Q-LEARNING ……………………………………………………………….………30 3.1 Q-LEARNING IN MARKOVIAN ENVIRONMENTS……………………..……....………….30 3.2 Q-LEARNING FUNCTION OVERVIEW…………………..…………..……………………...31 3.3 APPLICATION IN IOT NETWORKS……………………………..…..……...………………..32 3.4 REWARD CALCULATIONS IN IOT ENVIRONMENTS…………………………..………...32 3.5 Q-LEARNING ALGORITHM………………………..…….......................................................33 vii  3.5.1 INFLUENCE OF LEARNING RATE ON ESTIMATION ADJUSTMENTS……………..34 3.5.2 IMPACT ON LEARNING BEHAVIOR…………………………………………………..34 4 PROPOSED METHODOLOGY EQL-BASED DQMAC………………………...35 4.1.PROPOSED EQL DQMAC……………………………………………………...……………...36 4.2 JOINT CONTENTION TREE STRUCTURE………….……………………………………….37 4.3.MULTIPARALLEL CHANNEL MODEL……………………………………………………..38 4.4.OPERATION OF THE PROPOSED MECHANISM…………………………………………..40 4.5.DETAIL PRINCIPLES OF ENHANCED Q-LEARNING……………………………………..40 4.5.1 ACTION SELECTION……………………………………………………………………40 4.5.2.CONVERGENCE REQUIREMENTS……………………………………………………41 4.5.3. A PRIORI APPROXIMATE CONTROLLER…………………………………………..42 4.5.4. FORMULATION OF THE REWARD FUNCTION…………………………………….43 4.6. ALGORITHMS………………………………………………………………………………...44 4.6.1. ALGORITHM 1…………………………………………………………………………44 4.6.2. ALGORITHM 2…………………………………………………………………………45 4.6.3 ALGORITHM 3…………………………………………………………………………45 4.6.4 ALGORITHM 4…………………………………………………………………………46 5 RESULTS AND DISCUSSION.………………………………………………………...47 5.1 SIMULATION PARAMETERS…………………………………………….………................47 5.2 THROUGHPUT………….……….............................................................................................48 5.3 AVERAGE CONTENTIONS……………..................................................................................49 5.4 AVERAGE DELAY…………………..……………….……………………………………….50 5.5 ENERGY CONSUMPTION…….………...……………………………………………………52 6 CONCLUSION AND FUTURE WORK.….……...……………………………….55 6.1 CONCLUSION…………………………………………………………………………………..55 6.2 FUTURE WORK………………………………………………………………………………...56 7 REFERENCES ……………………………………………………………………...58 viii  LIST OF FIGURES FIG.1.1 CHANNEL ACCESSING TAXONOMY IN WSN………... …5 FIG.1.2 HIDDEN TERMINAL……………………………………......9 FIG.1.3 EXPOSED TERMINAL…………………………………….. …9 FIG.1.4 TOPOLOGY CONFIGURATIONS SUPPORTED BYIEEE 802.15.4…………………………………………………….. 11 FIG.1.5 SUPERFRAME STRUCTURE OF IEEE 802.15.4…………. ….12 FIG.1.6 HANDSHAKE BETWEEN COORDINATOR AND DEVICE WHEN THE DEVICE RETRIVES A PACKET… ….14 FIG.3.1 THE AGENT ENVIRONMENT INTERACTION IN MARKOV DECISION PROCESS………………………….. ….30 FIG.4.1 JOINT CONTENTION…………………………………………. …37 FIG.4.2 STRUCTURE OF MULTIPARALLEL CHANNEL……………... ….38 FIG.4.3 PROPOSED EQL-DQMAC WORKING………………………... ….39 FIG.4.4 FLOWCHART OF PROPOSED ALGORITHM…………….….44 FIG.5.1 COMPARISON OF SYSTEM THROUGHPUT OF IOT…... ….48 NODE IN TRADITIONAL DQMAC, QL-BASED DQMAC AND EQL-DQMAC IN IOT NETWORKS FIG.5.2 COMPARISON OF AVERAGE CONTENTION BEFORE...….50 SUCCESSFUL TRANSMISSION OF IOT NODE IN TRADITIONAL DQMAC, QL-BASED DQMAC AND EQL-DQMAC ix FIG.5.3 COPARISON OF MAC CONTENTION DELAY BEFORE..….52 SUCCESSFUL TRANSMISSION OF IOT NODE IN TRADITIONAL DQMAC, QL-BASED DQMAC AND EQL-DQMAC FIG.5.4 COMPARISON THE ENERGY CONSUMPTION FOR…... ….53 MAC CONTENTION IN TRADITIONAL DQMAC, QL- BASED DQMAC AND EQL-DQMAC IN IOT NETWORKS FIG.5.5 COMPARISON THE TOTAL ENERGY CONSUMPTION.. …54 PER BEACON INTERVAL IN TRADITIONAL DQMAC, QL-BASED DQMAC AND EQL-DQMAC IN IOT NETWORKS LIST OF TABLE TABLE.2.1 ADVANTAGE AND DISADVANTAGE OF VARIOUS….. ….29 PROTOCOL TABLE.4.1 INITIAL Q-VALUE TABLE FOR OUR PROPOSED EQL- ….42 BASED DQMAC SCHEME TABLE.5.1 SIMULATION PARAMETERS…………………………….. ….48 x CHAPTER-1 INTRODUCTION H A 1.1. Introduction to IOT Network Protocols: P T devices to The concept of the Internet of Things (IoT) revolves around the connectivity of sensor the internet in real-time. These devices interact with each other through networks,Enecessitating the establishment of standards and regulations governing data exchange. These regulations R are known as IoT Network Protocols. Presently, there exists a diverse array of IoT devices, each 1 requiring distinct protocols. The functionality of an IoT application determines its workflow or architecture, which typically comprises four layers: the Sensing layer, Network layer, Data processing layer, and Application layer. The Sensing layer encompasses all hardware components such as sensors, actuators, and chips responsible for gathering data. This layer interfaces with the subsequent layer, the network layer, through specific protocols. In the Network layer, devices communicate via various network protocols such as cellular, Wi-Fi, Bluetooth, Zigbee, etc. Data collected by IoT devices undergo processing in the Data processing layer, utilizing technologies like data analytics and machine learning algorithms. The processed data is then presented to users through web portals, applications, or interfaces provided by the Application layer, enabling direct interaction and visualization of IoT data. Designing protocols for IoT poses challenges due to the limited components of IoT devices, primarily comprising small batteries and sensors. Additionally, all operations, including constructing topological structures and address assignments, must be performed wirelessly, adding to the complexity. 1.1.1 IoT protocols must fulfill several critical requirements: 1. Simultaneous Communication:- They should facilitate communication among multiple devices concurrently to support the interconnected nature of IoT systems. 2. Communication Security:-Given the deployment of IoT in sensitive domains like healthcare, industries, and home surveillance, protocols must ensure robust security measures to protect data and privacy. 1|Page 3. Efficient Data Transport:- Efficient data transmission is essential for optimizing resource utilization and ensuring timely delivery of information across the IoT network. H A 4.Scalability:- Protocols should be scalable to accommodate the dynamic nature of IoT networks, P allowing for the seamless addition or removal of devices without compromising performance or reliability. T E When selecting a protocol, it's crucial to consider the specific environment and requirements it's designed for. Protocols vary in terms of range, data rates, power consumption, andR other factors. 1 For short-range communication with low data rates and low power consumption, Bluetooth is a viable option. Operating in the 2.4GHz frequency range, Bluetooth offers a range of 10m to 100m and supports data rates of up to 1Mbps. It accommodates two network topologies: point-to-point and mesh. Bluetooth is well-suited for transmitting small amounts of data to personal devices such as speakers, earphones, smartwatches, and smart shoes. Additionally, it can be employed in Smart Home applications, including alarms, HVAC systems, and lighting control. Zigbee, based on the IEEE802.15.4 standard, operates within the 2.4GHz frequency range, similar to Bluetooth. It offers a range of up to 100 meters with a maximum data rate of 250KBPS. Zigbee is ideal for transmitting small amounts of data over short distances and is particularly suitable for applications requiring high authentication and robustness. It supports multiple network topologies, including star, mesh, and cluster tree. Common applications of Zigbee include monitoring device health in industrial settings and facilitating smart home solutions. 6LoWPAN, short for IPv6 Low Power Personal Area Network, operates within the frequency range of 900 to 2400MHz, providing a data rate of 250KBPS. It supports star and mesh network topologies and is optimized for low-power consumption in personal area network environments. For short-range communication with high data rates, Wi-Fi (Wireless LAN) is an excellent choice. Wi-Fi offers high bandwidth, supporting data rates ranging from 54Mbps up to 600Mbps. It typically covers a range of 50m in a local area, but with private antennas, it can extend up to 30km. Wi-Fi enables easy connectivity of IoT devices, allowing them to share large amounts of data. It finds applications in smart homes, smart cities, offices, and various other settings. 2|Page For long-range communication with high data rates and low power consumption, two notable options are LoRaWAN and LTE-M: H LoRaWAN (Long Range Wide Area Network) provides extensive coverage, with a range of A approximately 2.5km to 15km. Although its data rate is relatively low, ranging from 0.3KBPS to P a maximum of 50KBPS, it can support numerous connected devices. LoRaWAN is utilized in applications such as Smart City deployments and Supply Chain Management. T E LTE-M (Long Term Evolution for Machines) is a type of LPWAN (Low Power RWide Area Network) that operates within a frequency range of 1.4MHz to 5MHz. It offers data 1 rates of up to 4MBPS and is often used in conjunction with cellular networks to provide enhanced security features. LTE-M is suitable for various IoT applications where reliable, high-speed data transmission is required. For long-range communication with low data rates and low power consumption, Sigfox is a viable option. Sigfox is designed for wide area coverage with minimal power consumption, aiming to connect billions of IoT devices. Operating at a frequency range of 900MHz, Sigfox offers a range spanning from 3km to 50km, with a maximum data rate of 1KBPS. Alternatively, for long-range communication with low data rates but high power consumption, cellular networks, including 2G, 3G, 4G, and 5G, are utilized. Cellular networks operate across various frequency ranges such as 900MHz and 1.8/1.9/2.1 GHz, providing a range of approximately 35km to 200km. The average data rates range from 35KBPS to 170KBPS. However, cellular networks consume relatively high power, which may not be suitable for all IoT devices. They are commonly employed in IoT applications like connected cars, where the benefits of widespread coverage and reliable communication outweigh the power consumption concerns. Now, let's delve into the intricacies of MAC protocols and their significance in the realm of IoT connectivity and communication. 1.2. Introduction of MAC Protocol: The MAC sub-layer constitutes an integral component within the data link layer protocol. It furnishes a mechanism for enabling multiple devices to access the channel when sharing the medium. In a wireless environment where multiple devices share the medium and communication is broadcasted, any transmission from one device is received by all other devices within its transmission range. This scenario may result in interference and collisions of frames when simultaneous transmissions from two or more devices occur at a single point. In 3|Page a scattered, dense, and rugged sensor field, sensor nodes typically communicate through multi- hop paths over the wireless medium. A MAC protocol governs the flow of communication over a common medium, establishing a foundational network structure for sensor nodes to interact. H Hence, it bestows nodes with self-organizing capabilities and endeavors to maintain network A unity by facilitating collision-free and errorless communication between senders and receivers. P Furthermore, the standard prerequisites for extending the lifespan of a WSN without necessitating power replacement or human intervention have stimulated theTcreation of innovative protocols across all layers of the communication stack. Yet, significantEbenefits can be attained within the data link layer, where the MAC protocol directly manages R radio 1 MAC operations, the most power-intensive aspect of resource-limited sensor nodes. Effective protocols employ the radio resource wisely to preserve its energy. Therefore, the MAC protocol contributes significantly to meeting key design goals of WSNs by delineating how nodes utilize radio, manage channel sharing, prevent collisions in correlated and broadcasting environments, promptly respond to inquiries, and prolong their operational lifespan. The IEEE 802 model provides details about the MAC (Media Access Control) layer protocols, which are specifically designed for local area networks (LANs). The MAC layer is responsible for controlling how devices access the shared physical medium (e.g., Ethernet cable, wireless channel) to transmit data. In the IEEE 802 model, the MAC layer is positioned above the Physical layer and below the LLC (Logical Link Control) layer. Three prominent MAC layer protocols as per the IEEE 802 model are: 1. CSMA/CD (Carrier Sense Multiple Access with Collision Detection): This is the MAC protocol used in traditional Ethernet networks. CSMA/CD allows multiple devices to share the same physical medium by listening for traffic before transmitting (Carrier Sense). If two devices transmit simultaneously, a collision occurs, which is detected (Collision Detection), and the devices retransmit after a random backoff period. 2. Token Bus: This is a MAC protocol used in token-bus networks, such as IEEE 802.4. In a token-bus network, devices are connected to a shared bus, and a small "token" frame is passed around to grant transmission rights. A device can transmit data only when it holds the token, preventing collisions. 3. Token Ring: 4|Page  This is a MAC protocol used in token-ring networks, such as IEEE 802.5. Devices are connected in a ring topology, and a small "token" frame circulates around the ring. H A device can transmit data only when it holds the token, ensuring only one device A transmits at a time. P 1.3. Classification of WSN MAC Protocols: T E Several MAC protocols have been successfully proposed to meet the stringent design R requirements of WSNs. Actually; these protocols depend on how protocol allows nodes to access the channel. We have classified WSN based MAC protocol as depicted in1Figurure( "The figures used in this section were referenced from." )into four categories as; contention based, scheduling based, channel polling based, and hybrid protocols. Fig.1.1. Channel Accessing taxonomy in wsn 1. Contention Based MAC Protocols: Contention-based MAC protocols indeed play a crucial role in wireless sensor networks (WSNs) due to their simplicity, flexibility, and suitability for event-driven applications. Here's a breakdown of the key points mentioned in the excerpt: i. Acquiring Channel Access:-Nodes in contention-based MAC protocols compete with their neighbors to access the channel for transmission. Before starting data transmission, a node senses the carrier to check if the channel is idle. If the channel is idle, the node starts transmission; otherwise, it defers transmission for a random period using a backoff algorithm. ii. Reduced Resource Consumption:-Contention-based MAC protocols are preferred for 5|Page  event-driven WSN applications because they reduce processing resource consumption. These protocols do not require clustering or topology information, allowing each node to independently contend for channel access without coordinating frame exchanges. H iii. Flexibility and Scalability:- Contention-based MAC protocols are A flexible and dynamic, making them suitable for networks of varying scales. They do Pnot rely on centralized coordination and can adapt to changes in network conditions.T E iv. Challenges and Issues:-Despite their benefits, contention-based MAC protocols face R challenges such as hidden and exposed terminals, which can lead to collisions, 1 overhearing, idle listening, and reduced throughput. These issues arise when nodes contend for channel access independently, leading to inefficient utilization of the channel and decreased network performance. v. Synchronized Contention Times:- Some MAC protocols synchronize the contention times of nodes according to a schedule. At each periodic interval, neighboring nodes wake up simultaneously to exchange packets. This synchronization helps in reducing collisions and improving overall network efficiency. 2. Channel Polling Based MAC Protocols: The channel polling scheme, also known as preamble sampling or Low Power Listening (LPL), involves sending preamble data packets with extra bytes by nodes. The preamble serves to ensure that the destination node detects radio activity and wakes up before receiving the actual payload from the sender. Here's a breakdown of how the channel polling scheme works: i. Preamble Transmission:- Nodes send a preamble over the channel to alert potential receivers of radio activity. The preamble is transmitted before the actual data payload, allowing receivers to wake up and listen for incoming packets. ii. Receiver Wake-Up:- Upon detecting radio activity from the preamble, the receiver node wakes up its radio to receive data packets. If no radio activity is detected, the receiver node goes back to sleep mode until the next polling interval. iii. Check Interval Duration:- The receiver periodically checks for radio activity during the check interval duration, which lasts until the preamble is sent. This ensures that the receiver remains awake long enough to detect incoming packets. 6|Page  iv. Synchronization and Clustering:- Unlike other MAC protocols that rely on synchronized active/sleep schedules, channel polling schemes do not require H synchronization, scheduling, or clustering among nodes. Each node independently A wakes up to listen for incoming transmissions based on the presence of preambles. P v. Combination with ALOHA:- The combination of ALOHA with preamble T sampling is a common example of the extended preamble-based channel polling E scheme. This approach helps in reducing energy consumption by allowing nodes to sleep when no R radio activity is detected. 1 vi. Berkeley MAC (BMAC) Protocol:- The Berkeley MAC protocol (BMAC) is an example of a channel polling scheme where the channel polling method is referred to as Low Power Listening (LPL). BMAC utilizes extended preambles to enable low- power listening and efficient communication in WSNs. 3. Scheduling Based MAC Protocols: Scheduling-based MAC schemes allocate collision-free links between neighboring nodes during the initialization phase. These schemes divide the system time into slots and assign them to nodes, controlling participant authorization on resources with regular timing. Here's a breakdown of how scheduling-based MAC schemes work and their advantages and challenges: i. Time Division Multiplexing (TDMA):- TDMA schemes divide system time into slots and allocate them to neighboring nodes. The schedule, regulated by a central authority, controls resource access. Nodes do not contend with neighbors and can only access their allocated time slots. TDMA offers advantages such as minimum collisions, reduced overhearing, avoidance of idle listening, and predictable end-to-end delay. ii. Advantages:- Minimum collisions and reduced overhearing. Predictable end-to-end delay. Bounded and predictable queuing delay. iii. Challenges:- High average queuing delay due to waiting for allocated time slots. Overhead and extra traffic. Lack of adaptability. Reduced scalability. 7|Page  Difficulty in allocating conflict-free TDMA schedules. Inability for peer-to-peer communication without involving the central authority. H iv. Representative Protocols:- A DRAND:- A distributed slot assignment scheme that overcomes P the difficulty of obtaining global topology information. T PACT:- Another distributed slot assignment scheme aimed at large Enetworks. TRAMA:- A protocol that obtains local topology and interference information R at each node. 1 Flow-Aware Medium Access (FLAMA):- Derived from TRAMA, optimized for periodic monitoring applications to avoid overhead. Low-Complexity Slot Selection Mechanisms:- Simplified schemes like LMAC aim to achieve good energy efficiency by reducing radio state transitions and protocol overhead. Adaptive Information-centric LMAC (AILMAC):- AILMAC tailors slot assignment to actual traffic needs, addressing the fixed frame length issue of LMAC. Joint Interconnect Protocol:- Addresses maximum throughput and fair rate allocation in WSNs with consideration for slot reuse-based TDMA. 4. Hybrid MAC Protocol: Hybrid MAC protocols offer a promising approach to leverage the advantages of different MAC schemes while mitigating their individual weaknesses. By combining synchronized schemes, such as TDMA (Time Division Multiple Access), with asynchronous schemes, like CSMA (Carrier Sense Multiple Access), hybrid protocols aim to achieve a balance between efficiency, scalability, and complexity. One notable example of a hybrid MAC protocol is the Zebra MAC (Z-MAC) protocol. Z-MAC combines the strengths of TDMA and CSMA, leveraging the deterministic access provided by TDMA for predictable communication scheduling while also benefiting from the flexibility and adaptability of CSMA. By dynamically switching between TDMA and CSMA modes based on network conditions, Z-MAC optimizes both energy efficiency and throughput. Another example is the Scheduled Channel Polling MAC (SCP-MAC), which integrates scheduled channel polling with contention-based access methods. SCP-MAC allows nodes to operate in a synchronized manner, similar to TDMA, while also supporting contention-based 8|Page access during periods of low traffic. This hybrid approach enables SCP-MAC to achieve low- latency communication while maintaining scalability and adaptability. H A Similarly, the Funneling-MAC protocol combines TDMA-based scheduling with funneling P techniques to improve channel utilization and reduce contention overhead. By organizing nodes T into hierarchical structures and employing time-slotted channel access, Funneling-MAC optimizes resource allocation and minimizes collisions in large-scale networks. E R 1 While hybrid MAC protocols offer the potential for significant performance improvements, they may also introduce challenges related to scalability and complexity. Managing multiple working modes and coordinating the interaction between synchronized and asynchronous schemes can increase overhead and complicate protocol design. However, with careful design and optimization, hybrid MAC protocols can effectively address the diverse requirements of IoT applications and improve overall network performance. Fig.1.2. Hidden terminal The hidden terminal problem occurs when two nodes, such as nodes A and C in the example, are within range of a common receiver, node B, but are out of range of each other. In this scenario, node C may mistakenly sense the medium as idle when node A is transmitting to node B, leading to a collision at node B because node A cannot detect the transmission from node C. This problem arises due to the inability of nodes to detect transmissions from hidden nodes, resulting in collisions at the receiver. 9|Page  Fig.1.3. Exposed terminal On the other hand, the exposed terminal problem occurs when a node, such as node C in the H example, refrains from transmitting even though the medium is idle because it mistakenly A assumes that the transmission from node B to node A will interfere with its transmission to node D. In reality, there would be no interference, and node C unnecessarily delays its P transmission, leading to decreased network efficiency. This problem arises due to nodes being T overly cautious about potential interference from transmissions outside their range. E R Both the hidden terminal problem and the exposed terminal problem can significantly impact 1 reduced the performance and reliability of a wireless network, leading to increased collisions, throughput, and inefficient use of the available bandwidth. Effective MAC protocols should aim to mitigate these problems through techniques such as carrier sensing, clear channel assessment, and efficient channel access mechanisms. 1.4. Characteristics of MAC Protocols: The prevalent attributes of MAC protocols can be encapsulated as follows: Throughput:-Throughput refers to the speed at which messages are processed. It can be quantified in messages or symbols per second, although the predominant unit of measurement is bits per second. The objective is to optimize this rate. Transmission delay:- Transmission delay refers to the duration during which a solitary message remains within the MAC protocol. Fairness:- Fairness in a MAC protocol is defined by its ability to distribute the channel among competing nodes based on predetermined fairness criteria. Scalability:-Scalability refers to the capability of a communication system to maintain its performance standards regardless of the network's size and the number of nodes vying for resources. Robustness:-Robustness encompasses reliability, availability, and dependability in its composition. Stability:-Stability refers to the protocol's capability to effectively manage variations in traffic 10 | P a g e load over an extended duration. 1.5. Goals of MAC Protocols: H Avoid Overhearing: Refrain from needlessly receiving packets meant for other nodes. Reduce Idle Listening: Avoid remaining active to receive data when thereAis no sender. Decrease Active Time: Minimize the duration of active listening. P Prevent Packet Collisions: Take measures to eliminate collisions betweenTpackets. Reduce Control Packet Overhead: Minimize the amount of overhead E caused by control packets. R Avoid Buffer Overflow: Prevent overflow in the buffer to maintain efficient operation. 1 1.6. IEEE 802.15.4 MAC The IEEE introduced the 802.15.4 MAC standard for Wireless Personal Area Networks (WPANs), incorporating a duty cycle mechanism that allows for adjustable sizes of active and inactive periods during PAN formation. Network Architecture and Types/Roles of Nodes: The IEEE 802.15.4 MAC combines both the schedule-based and contention-based protocols and supports two network topologies, star and peer-to-peer as shown in Figure Fig.1.4. Topology configurations supported by IEEE 802.15.4 1.6.1. Applications of IEEE 802.15.4 1. Wireless Sensor Networks 2. Home Automation Systems 3. Home Networking Solutions 4. Interconnecting Devices with PCs 5. Home Security Systems 6. Industrial Automation 11 | P a g e 7. Smart Grids for Energy Management 8. Healthcare Monitoring Systems 9. Environmental Monitoring and Control H 10. Asset Tracking and Management A There are two distinct peer-to-peer topology variants. The first, termed a cluster-tree network, P has found widespread application in ZigBee technology. The second variant, referred to as a mesh network, is extensively utilized within the IEEE 802.15 WPAN Task GroupT 5 (TG5). The standard delineates two node types: the Full Function Device (FFD) and the ReducedE Function Device (RFD). FFD nodes possess the flexibility to assume three distinctR roles: PAN coordinator, coordinator, and device. Conversely, RFDs are limited to operating 1 solely as devices. Under all network conditions, devices are required to be associated with a coordinator. Multiple coordinators can function within either a peer-to-peer or star topology, with one coordinator assuming the role of the PAN coordinator. The star topology is better suited for delay-critical applications and small network coverage due to its efficient communication structure. In contrast, the peer-to-peer topology is more applicable for large networks with multi-hop requirements, albeit at the expense of higher network latency. Moreover, the standard delineates two modes for data exchange: beacon mode and non-beacon mode. Beacon mode offers synchronization measures to networks, whereas non-beacon mode provides asynchronous features. 1.6.2. Superframe Structure: The beacon mode of IEEE 802.15.4 MAC introduces a superframe structure to organize channel access and data exchanges. The structure figure 1.6, comprises two primary periods: the active period and the inactive period. The active period is subdivided into 16 time slots, typically beginning with the transmission of the beacon frame in the first slot. Following this, two additional segments, the Contention Access Period (CAP) and the Contention-Free Period (CFP), utilize the remaining slots. The CFP, also referred to as Guaranteed Time Slots (GTS), can employ up to 7 time slots. 12 | P a g e  Fig.1.5. Superframe structure of IEEE 802.15.4 The duration of both the active and inactive periods, as well as the length of an individual time slot, are configurable and vary based on traffic conditions. Data transmissions can take place H either in the Contention Access Period (CAP) or in Guaranteed Time Slots (GTS). In CAP, data communication employs slotted CSMA-CA (Carrier Sense Multiple Access with A Collision Avoidance), while in GTS, nodes are allocated fixed time slots dedicated P to data communication. To achieve energy-efficient operations in IEEE 802.15.4 MAC,Tthe strategy Eneither data involves putting nodes to sleep during the inactive period, as well as when there is to transmit nor any data to fetch from the coordinator. Nevertheless, the energy burden falls R primarily on the coordinator, which must remain active throughout the entire active period. 1 1.6.3. GTS Management: The coordinator assigns Guaranteed Time Slots (GTS) to devices only upon receiving appropriate request packets during the Contention Access Period (CAP). A flag within the request indicates whether the requested time slot is for transmission or reception. In a transmit slot, the device sends packets to the coordinator, while in a receive slot, data flows in the opposite direction. Another field in the request specifies the desired number of contiguous time slots in the GTS phase. The coordinator responds to the request packet in two steps: Firstly, an immediate acknowledgment packet confirms the receipt of the request packet by the coordinator, but contains no information regarding the success or failure of the request. Upon receiving the acknowledgment packet, the device must monitor the coordinator's beacons for a specified duration known as a GTSDescPersistenceTime. When the coordinator has adequate resources to allocate a GTS to the node, it inserts an appropriate GTS descriptor into one of the subsequent beacon frames. This GTS descriptor includes the short address of the requesting node and details about the number and position of the time slots within the GTS phase of the superframe. A device can utilize its allocated slots each time they are announced by the coordinator in the GTS descriptor. If the coordinator lacks sufficient resources, it generates a GTS descriptor for an (invalid) time slot zero, indicating the available resources in the descriptor's length field. Upon receiving such a descriptor, the device may consider renegotiation. If the device does not receive a GTS descriptor within the specified GTSDescPersistenceTime after sending the 13 | P a g e request, it assumes that the allocation request has failed. A GTS is allocated to a device regularly until it is explicitly deallocated. The device can request deallocation by sending a H special control frame. Upon sending this frame, the device should refrain from using the A allocated slots further. P Alternatively, the coordinator can trigger deallocation based on certain criteria.T Specifically, the coordinator monitors the usage of the time slot. If the slot remains unused forEa certain number of superframes, it is deallocated. The coordinator signals deallocation to Rthe device by generating a GTS descriptor with start slot zero. 1 1.6.4. Data Transfer: When a device intends to transmit a data packet to the coordinator and possesses an allocated transmit GTS, it awakens just before the time slot initiation and promptly sends its packet without conducting carrier-sense or other collision-avoidance procedures. However, this action is feasible only if the entire transaction, including the data packet, an immediate acknowledgment from the coordinator, and appropriate InterFrame Spaces (IFSs), can fit within the allocated time slots. When a device needs to transmit a data packet to the coordinator and has an allocated transmit GTS, it sends the packet without carrier-sensing operations if the entire transaction, including the packet, coordinator's immediate acknowledgment, and InterFrame Spaces (IFSs), fits within the allocated time slots. If not or if the device lacks allocated slots, it transmits during the Contention Access Period (CAP) using slotted CSMA. 14 | P a g e  Fig.1.6. Handshake between coordinator and device when the device retrieves a packet The coordinator acknowledges the packet. Conversely, if the coordinator wants to send a packet to the device and the device has an allocated receive GTS that accommodates the packet cycle, H the coordinator transmits it in the allocated slot without further coordination.AThe device acknowledges the packet, following a handshake protocol. The coordinator signalsP a buffered T frame, packet by including the device's address in the pending address field of the beacon prompting the device to send a data request packet during CAP. Upon receiving E this, the coordinator acknowledges and sends the data packet, with the device preparing for its reception. R If unsuccessful, the device retries during subsequent superframes, optionally powering off its 1 transceiver until the next beacon. 1.6.5. Slotted CSMA-CA Protocol: When nodes need to transmit data or management/control packets during the Contention Access Period (CAP), they utilize a slotted CSMA protocol. However, this protocol does not incorporate provisions to address hidden-terminal situations. For instance, there is no RTS/CTS handshake implemented. In order to diminish the likelihood of collisions, the protocol integrates random delays, thereby classifying it as a CSMA-CA protocol (Carrier Sense Multiple Access with Collision Avoidance). The time slots within the Contention Access Period (CAP) are further divided into smaller segments known as backoff periods. Each backoff period spans a length equivalent to 20 channel symbol times, and only these backoff periods are taken into consideration by the slotted CSMA-CA protocol. 1.7. DQ Mechanism:- The DQ protocol, pioneered by , introduces a novel approach where active devices contend in contention slots before proceeding with data transmission. Central to this protocol is the role of a coordinator, tasked with disseminating crucial feedback information concerning the state of each contention slot. This feedback assists in organizing devices into one of two logical queues: the collision resolution queue (CRQ) and the data transmission queue (DTQ). Upon receiving feedback, devices are strategically placed based on the observed state of contention. If collisions are detected during contention, devices are segmented and queued into the CRQ for subsequent resolution. Conversely, if no collisions occur, devices are queued into the DTQ, awaiting a collision-free transmission opportunity. This dynamic queue management system optimizes network efficiency by mitigating collisions and ensuring smoother data transmission processes. 15 | P a g e In the context of , the gateway assumes the pivotal role of a coordinator within the DQ protocol framework. All devices operating within this environment are required to operate in H class B mode to facilitate the reception of feedback packets at predefined periodic intervals. A P This periodic feedback mechanism enables devices to stay synchronized with the coordinator's instructions and adapt their operations accordingly. Furthermore, each superframe T is meticulously divided into three distinct segments: the contention window, data E slot, and R with the feedback slot. These segments serve specific purposes within the protocol's operation, contention window facilitating contention resolution, the data slot enabling data 1 transmission, and the feedback slot facilitating the broadcast of crucial feedback information. This structured division of the superframe ensures efficient utilization of network resources and enhances the overall performance and reliability of the DQ protocol implementation. The research paper presents PDQRAP (Prioritized Distributed Queueing Random Access Protocol), a prioritized variant of the DQRAP protocol designed to facilitate efficient transport of multimedia traffic over shared mediums such as LANs, MANs, and WANs. PDQRAP introduces two distinct formats: extra-bit and extra-slot, enabling nodes to differentiate between high-priority and normal-priority traffic. This prioritization scheme aims to enhance throughput and reduce delay for high-priority traffic segments, with observed improvements ranging from 20% to 30% under 90% load conditions, experiencing one-third to one-half the delay compared to normal-priority traffic. However, while PDQRAP offers significant benefits, it also poses several potential limitations. These include increased overhead resulting from prioritization mechanisms, concerns regarding fairness and potential starvation of lower-priority traffic, heightened protocol complexity, and scalability challenges as network size or priority levels increase. Despite these considerations, PDQRAP presents a promising approach to effectively accommodate multimedia services with varying priorities over shared network infrastructures. In research paper addresses the critical issue of sleep/wake-up scheduling, vital for conserving the limited energy resources of sensor nodes. By maximizing network lifetime while ensuring efficient packet delivery, sleep/wake-up scheduling plays a pivotal role in WSNs. The paper explores the use of duty-cycling techniques, where nodes alternate between active and sleep modes based on a predefined schedule. However, this approach presents a trade-off between energy savings and packet delivery delay, as nodes risk missing transmissions during sleep. To mitigate these trade-offs, researchers have proposed self- adaptive sleep/wake-up scheduling algorithms, enabling nodes to autonomously determine 16 | P a g e their operational mode in each time slot using reinforcement learning techniques. While promising, designing such mechanisms poses challenges, as nodes must balance energy conservation with network connectivity and responsiveness, necessitating sophisticated H algorithm design and decision-making processes. A The paper explores the concept of optimizing system throughput in high-traffic P network scenarios by employing dormant states for certain nodes, which intermittently T awaken to transmit data, potentially reducing collisions and enhancing overall throughput. Introducing E the Q-learning-based distributed queuing medium access control (QL-DQMAC) protocol for R Internet of Things (IoT) networks, this work aims to determine the optimal number of 1 contending IoT nodes in each contention period. Each node independently calculates its active rate using a Q-learning algorithm and decides whether to be active or enter a dormant state in the next contention period based on this rate. By optimizing the number of active nodes, the QL-DQMAC protocol effectively reduces collision probabilities, leading to lower energy consumption and decreased delays associated with the medium access control (MAC) contention process. Comparative analysis against other DQMAC approaches underscores the superior performance of the QL-DQMAC protocol in IoT network environments. In the realm of Software-Defined Networking (SDN), the paper delves into the remarkable programmability and extensive capabilities offered by SDN for network management operations. The centralized intelligence of the controller empowers it to adapt routing policies dynamically based on application requirements. To further augment these capabilities, the integration of Artificial Intelligence (AI) tools endows the controller with the ability to autonomously reconfigure the network in response to changing conditions. The paper specifically focuses on deploying a Q-learning algorithm for optimizing routing, with a primary emphasis on minimizing latency. The authors investigate the impact of exploration-exploitation strategies on the algorithm's performance and propose enhancements to optimize its efficacy within the target environment. These enhancements include incorporating a congestion- avoidance mechanism during the exploration phase and introducing a novel strategy based on the Max-Boltzmann Exploration method (MBE), which amalgamates traditional ε-greedy and softmax strategies. The empirical results underscore the superiority of the MBE strategy, coupled with the congestion-avoidance mechanism, demonstrating superior performance in terms of average latency, convergence time, and computational efficiency when compared to ε-greedy, ε-decay, and softmax strategies. 1.8. Machine learning:- 17 | P a g e Machine learning, with its ability to extract patterns and insights from large datasets, has indeed become a valuable tool in addressing complex problems in communication networks. Here are some key ways in which machine learning is applied in this domain: H 1. Network Optimization:- Machine learning algorithms can optimize various aspects of A communication networks, such as routing, resource allocation, and spectrum P management. By analyzing network traffic patterns and performance metrics, machine Tthroughput, learning models can dynamically adjust network parameters to maximize minimize latency, and optimize resource utilization. E R 2. Anomaly Detection and Security:- Machine learning techniques are used to detect 1 anomalies and security threats in communication networks. By learning patterns of normal network behavior, machine learning models can identify deviations indicative of malicious activities, such as network intrusions, denial-of-service attacks, or anomalous traffic patterns. 3. Quality of Service (QoS) Improvement:- Machine learning algorithms can enhance the quality of service in communication networks by predicting network congestion, optimizing traffic routing, and prioritizing critical data packets. By analyzing historical network performance data, machine learning models can dynamically adjust network configurations to meet QoS requirements and ensure reliable and efficient data transmission. 4. Resource Management and Allocation:- Machine learning is applied to optimize the allocation of network resources, such as bandwidth, power, and computing resources. By learning from past usage patterns and user behavior, machine learning models can allocate resources more efficiently, ensuring fair distribution and minimizing wastage. 5. Network Planning and Deployment:- Machine learning techniques are used in network planning and deployment to optimize the placement of network nodes, antennas, and infrastructure components. By analyzing geographical data, terrain information, and user demographics, machine learning models can determine optimal locations for network deployment, maximizing coverage and minimizing interference. 1.8.1. The three main types of machine learning algorithms and their applications in communication networks: 1. Supervised Learning:- This involves training a model using labeled data to predict output for new, unseen data. In communication networks, supervised learning is used for tasks such as network traffic classification, link quality prediction, and resource allocation, where the goal is to make accurate predictions based on known input-output 18 | P a g e  relationships. 2. Unsupervised Learning:- This type of learning aims to discover hidden patterns and structures in data without labeled information. Algorithms like K-means clustering and H DBSCAN are commonly used in communication networks for tasks such as anomaly A detection and network topology identification, where the goal is to identify clusters or groups within the data based on similarities. P T 3. Reinforcement Learning (RL):- RL involves an agent learning to makeE decisions by interacting with an environment and receiving feedback in the form of Rrewards or penalties. In communication networks, RL has been applied to problems 1 such as dynamic spectrum access, power control, and routing optimization, where the agent learns to take actions that maximize cumulative rewards over time the essence of reinforcement learning (RL) well! RL indeed revolves around an agent learning to navigate an environment by taking actions and receiving feedback, typically in the form of rewards or penalties, to maximize long-term cumulative rewards. In the context of communication networks, RL has shown promise in addressing various challenges, including: 1. Dynamic Spectrum Access:- RL algorithms can be used to adaptively allocate available spectrum resources to users or devices in real-time based on changing network conditions and demand. By learning optimal spectrum utilization strategies, RL agents can improve spectrum efficiency and minimize interference. 2. Power Control:- RL techniques can optimize power control strategies in wireless networks to enhance energy efficiency, extend battery life, and improve overall network performance. Agents learn to adjust transmission power levels dynamically based on channel conditions and traffic patterns to achieve desired objectives. 3. Routing Optimization:- RL algorithms can optimize routing decisions in communication networks by learning to select the most efficient paths for data transmission. Agents learn to balance factors such as latency, throughput, and energy consumption to optimize overall network performance. By leveraging RL techniques in these areas, communication networks can become more adaptive, resilient, and efficient, ultimately enhancing user experience and enabling the successful deployment of emerging technologies like IoT and 5G. 19 | P a g e 1.9. Introduction to Qlearning:- Q-learning is a fundamental reinforcement learning technique that has been widely applied in H various optimization problems, including those encountered in communication networks. A Here's a breakdown of its key characteristics and applications: P 1. Action-Value Function:- Q-learning learns an action-value function (Q-function)T that E state. This estimates the expected cumulative reward for taking a particular action in a specific R potential function guides the agent's decision-making process by associating actions with their long-term benefits. 1 2. Simplicity:- Q-learning is known for its simplicity and elegance. The algorithm iteratively updates Q-values based on observed rewards, requiring minimal prior knowledge about the environment. This simplicity makes it easy to implement and understand, even in complex network scenarios. 3. Convergence Guarantees:- Q-learning is guaranteed to converge to the optimal action- value function under certain conditions, ensuring that the agent eventually learns the best policy for maximizing rewards. This convergence property provides confidence in the effectiveness of Q-learning algorithms. 4. Handling Unknown Environments:- Q-learning excels in environments with unknown dynamics or complex state-action spaces. The agent explores the environment by trying different actions and learning from the observed rewards, gradually refining its policy over time to adapt to changing conditions. In communication networks, Q-learning has been successfully applied to various optimization problems: Resource Allocation:- Q-learning can optimize resource allocation strategies by dynamically allocating network resources such as bandwidth, spectrum, or power to users or applications. The agent learns to balance competing objectives such as throughput, delay, and fairness to improve overall network performance. Medium Access Control (MAC):- Q-learning algorithms can optimize MAC protocols by learning to make efficient decisions regarding channel access, contention resolution, and transmission scheduling. This helps mitigate issues like collisions, 20 | P a g e  latency, and energy consumption in wireless communication systems. Routing:- Q-learning techniques can optimize routing decisions by learning to select the most appropriate paths for data transmission based on network conditions and H performance objectives. Agents learn to adaptively route traffic to minimize congestion, A packet loss, and end-to-end delay. P By leveraging Q-learning in these areas, communication networks can become more adaptive, T efficient, and resilient, ultimately enhancing the quality of service for users and applications. E Absolutely, machine learning, particularly reinforcement learning (RL), can playR a crucial role in enabling intelligent actions and adaptive behaviors in IoT networks. Here's how 1 RL can be applied to IoT nodes for accessing the communication channel under dynamic traffic load. 1. Dynamic Channel Access:- In IoT networks, the communication channel often experiences varying levels of congestion and interference due to the dynamic nature of IoT traffic. RL algorithms can enable IoT nodes to dynamically adjust their channel access strategies based on real-time network conditions. By learning from past experiences and feedback, IoT nodes can optimize their channel access decisions to maximize throughput, minimize latency, and ensure fair resource allocation. 2. Adaptive Behavior:- RL allows IoT nodes to adapt their channel access policies in response to changes in network topology, traffic patterns, and environmental conditions. For example, when congestion is detected, IoT nodes can learn to back off and retransmit packets at a later time to avoid collisions and improve overall network efficiency. Similarly, RL algorithms can help IoT nodes prioritize critical traffic and adjust transmission parameters dynamically to meet quality-of-service (QoS) requirements. 3. Optimization of Resource Utilization:- RL-based channel access schemes can optimize the utilization of available resources, such as bandwidth, spectrum, and energy, by intelligently allocating them to different IoT devices and applications. By learning to balance conflicting objectives, such as maximizing throughput while minimizing energy consumption, RL-equipped IoT nodes can achieve more efficient resource utilization and prolong network lifetime. 4. Self-Learning and Adaptation:- RL enables IoT nodes to learn from interactions with the environment and autonomously improve their channel access strategies over time. As network conditions evolve and new challenges arise, RL-equipped IoT nodes can continuously adapt and refine their behavior without human intervention. This self- 21 | P a g e  learning capability is particularly valuable in dynamic and unpredictable IoT environments where traditional rule-based approaches may fall short. H The proposed EQL-DQ algorithm in this study falls under the category of reinforcement A learning, specifically leveraging the Q-learning technique. By combining Q-learning with distributed queueing along with joint contention and multiparallel channel model, PEQL-DQ aims to adaptively manage the dynamic conditions of IoT networks and provide T differentiated services to support the diverse quality-of-service needs of various IoT applications. E 1.10. Introduce EQL-DQ : R 1 The EQL-DQ, an algorithm aimed at enhancing the efficiency of IoT networks. By integrating Q-learning, which leverages reinforcement learning with an enhanced reward function, with distributed queuing techniques, EQL-DQ facilitates parallel execution of contention resolution queues (CRQ) and data transfer queues (DTQ). This joint approach works synergistically to improve overall network performance, contention management, and data transfer capabilities. EQL-DQ dynamically adjusts node contention, proficiently managing collisions and transmissions. The algorithm prioritizes balanced optimization of throughput, delay, and energy efficiency, thereby addressing challenges present in existing MAC protocols for IoT applications. The EQL-DQ algorithm offers several key features that contribute to its efficacy in enhancing IoT network efficiency: i. Reinforcement Learning-based Optimization:- EQL-DQ adopts a reinforcement learning-based approach to adaptively optimize the number of contending nodes. Leveraging Q-learning with an enhanced reward function, the algorithm dynamically adjusts the number of contending nodes in each contention period. This adaptive optimization aims to maximize throughput while minimizing energy consumption, thus improving overall network efficiency. ii. Distributed Queueing Mechanism:- EQL-DQ incorporates a distributed queueing mechanism that enables parallel execution of contention resolution queues (CRQ) and data transfer queues (DTQ). This mechanism facilitates the joint contention resolution in the CRQ and parallel multichannel data transfer in the DTQ, leading to improved performance in contention management and data transfer capabilities. By concurrently handling contention resolution and data transmission tasks, EQL-DQ enhances network efficiency and resource utilization. 22 | P a g e CHAPTER-2 H Literature Review A P In , the paper presents RC-SFAMA, a MAC protocol specifically designed for Tunderwater acoustic networks, with the goal of overcoming the limitations associated with the prevalent E slotted-FAMA protocol. RC-SFAMA integrates fundamental concepts from slotted-FAMA R while introducing a novel time slot mechanism to mitigate data collisions, particularly in dense 1 network environments. It also addresses the challenge of multiple RTS (Request-to-Send) attempts in dense networks through an RTS competition mechanism, thereby enhancing throughput efficiency. The protocol offers several advantages, including improved throughput efficiency, especially in dense network scenarios, reduced energy consumption by minimizing unsuccessful transmission attempts, and enhanced applicability in dense network deployments. However, it is noteworthy that RC-SFAMA exhibits certain limitations, such as increased complexity due to the RTS competition mechanism, the requirement for device synchronization, and sensitivity to fluctuating network conditions. Despite these limitations, RC-SFAMA represents a significant advancement in MAC protocols for underwater acoustic networks, offering promising solutions to address the unique challenges posed by such environments. The paper provides a comprehensive exploration of alternative Medium Access Control (MAC) protocols and techniques, aiming to address the limitations of the widely used ALOHA protocol in wireless communication systems. It delves into contention-based protocols like Carrier Sense Multiple Access with Collision Avoidance (CSMA/CA), emphasizing the utilization of carrier sensing and backoff procedures to enhance throughput by mitigating collisions. Moreover, reservation-based protocols such as Time-Division Multiple Access (TDMA) and Frequency-Division Multiple Access (FDMA) are examined, offering deterministic access and improved Quality of Service (QoS) support. The paper also investigates hybrid protocols, such as CSMA/CA with reservation, to leverage the advantages of both contention-based and reservation-based approaches. Furthermore, reinforcement learning-based MAC protocols are discussed for their potential to dynamically adapt to varying network conditions and traffic demands. These alternative MAC solutions offer various benefits, including enhanced throughput, spectral efficiency, QoS support, and adaptability to diverse communication requirements, such as those encountered in IoT and 5G contexts. However, the paper acknowledges that implementing and deploying these alternative MAC 23 | P a g e solutions may introduce additional complexity and overhead compared to the simpler ALOHA protocol, potentially hindering their widespread adoption. H A In paper , a comprehensive distributed queue-based random access framework is proposed P to address the significant challenges associated with supporting massive Machine-Type Communications (mMTC) within LTE/LTE-A networks featuring mixed-typeTtraffic. The central idea revolves around employing a distributed queueing (DQ) Emechanism, complemented by MAC layer load estimation, to effectively manage contention R among massive Machine-Type Devices (MTDs) and enhance delay performance while 1 minimizing disruptions to the LTE access procedure and air interface. A notable advantage of this framework is its ability to dynamically prioritize access by leveraging congestion information from the DQ process without incurring additional signaling overhead. Simulation results demonstrate that the proposed framework surpasses the 3GPP baseline Access Class Barring (ACB) scheme in terms of access delay and energy consumption, particularly in scenarios where all MTDs hold equal importance. Furthermore, in scenarios where MTDs possess different priorities, the framework can be fine-tuned to achieve lower delays across all classes and reduced overall energy usage compared to both the baseline and dynamic ACB solutions, particularly in bursty massive access scenarios. However, a potential limitation lies in the necessity for optimized parameter settings to fully harness the capabilities of the proposed DQ and prioritization techniques across a spectrum of diverse mMTC traffic patterns. In , the paper addresses the shortcomings of the slotted ALOHA random access procedure utilized for Machine-to-Machine (M2M) or Machine-Type Communications (MTC), particularly concerning high user loads and limited available preambles. It introduces an enhancement to the 3GPP Extended Access Barring (EAB) mechanism by proposing a static and dynamic expansion of the contention space, aiming to mitigate collisions and increase the success probability to meet stringent delay and energy constraints. The primary advantage of this contention space expansion approach is its ability to outperform the existing 3GPP EAB mechanism in terms of access delay, collision rate, and energy efficiency, particularly in MTC scenarios with a massive number of devices. However, a potential limitation lies in the possibility of static expansion leading to inefficient resource utilization during low device loads, while dynamic expansion could introduce additional signaling overhead. The paper employs both an analytical model and simulations to assess the performance impact of the proposed contention space expansion on the 3GPP EAB mechanism across varying device loads, providing insights into its efficacy in real-world deployment scenarios. 24 | P a g e In their paper , the authors present an enhanced SARSA(λ) reinforcement learning algorithm specifically tailored for wireless communication systems. SARSA(λ) is an extension H of the traditional SARSA algorithm, leveraging eligibility traces to enhance learning efficiency A by considering long-term action consequences. When applied to wireless communication systems, the primary objective is to optimize key system performance metrics P such as T approach throughput, energy efficiency, or quality of service. The reinforcement learning enables adaptive decision-making based on interactions with the environment and Efeedback in the form of rewards or penalties. The benefits of the enhanced SARSA(λ) algorithm include R demonstrated superiority over traditional SARSA in terms of convergence speed and overall 1 system performance, adaptability to changing network dynamics, and potential for more efficient resource allocation. However, several challenges are noted, including increased computational complexity, the exploration-exploitation tradeoff inherent in reinforcement learning, sensitivity to initial conditions, scalability concerns for larger networks, and the absence of theoretical guarantees on convergence or optimality. Despite these challenges, the enhanced SARSA(λ) algorithm represents a promising approach for optimizing wireless communication systems by leveraging the power of reinforcement learning techniques. In the realm of Machine-Type Communications (MTC) and Internet of Things (IoT) environments, characterized by numerous devices contending for channel access, traditional single-channel ALOHA faces inherent limitations due to congestion, collisions, and reduced throughput. To address these challenges and accommodate the growing device population, multichannel ALOHA emerges as a promising solution. Multichannel ALOHA extends the ALOHA concept by allowing devices to transmit data over multiple channels randomly, thereby mitigating congestion and enhancing system throughput. Variations, such as segmenting channels into data and control channels, have been explored to analyze their throughput and stability properties. The adoption of multichannel ALOHA offers several benefits, including increased throughput, improved channel load balancing, and reduced potential delays and collisions. However, traditional multichannel ALOHA still faces challenges such as high collision rates under heavy channel loads, absence of channel sensing or exploration mechanisms, and the lack of means to estimate or adapt to the dynamic number of active devices. Therefore, further optimization is necessary to ensure efficient operation in MTC and IoT contexts. In , the paper by Wu et al. introduces a distributed queueing-based random access protocol tailored for LoRa (Long-Range) networks, a popular low-power wide-area network (LPWAN) technology extensively utilized in various Internet of Things (IoT) applications. The protocol 25 | P a g e aims to tackle the challenges associated with efficient channel access and collision avoidance in dense LoRa networks, where numerous devices contend for limited radio resources. The key concepts discussed include: Distributed Queueing: The protocol employs a distributed H queueing mechanism to coordinate channel access among LoRa devices. Each device maintains A a local queue to manage its transmission attempts. This approach, combined with a random P access scheme, reduces the likelihood of collisions and enhances overall network performance. T The protocol accounts for specific characteristics of LoRa networks, such as spread-spectrum modulation, limited orthogonal channels, and bursty IoT traffic. The distributedE queueing- based random access protocol for LoRa networks offers several benefits: ImprovedR Channel Access Efficiency: The distributed queueing mechanism enhances channel access1 coordination, reducing collisions and improving overall network throughput. Scalability: The protocol accommodates a large number of LoRa devices in dense networks, making it suitable for IoT applications with high device density. Reduced Latency: By prioritizing channel access, the protocol minimizes transmission latency experienced by LoRa devices. Energy Efficiency: The random access scheme and distributed queueing approach improve energy efficiency by reducing failed transmission attempts and unnecessary channel access. However, the protocol faces some potential limitations: Complexity: Implementing distributed queueing and associated decision-making processes may introduce additional complexity, impacting computational and memory requirements. Synchronization: The protocol assumes device synchronization, which may be challenging in decentralized network environments. Sensitivity to Network Dynamics: Performance may be sensitive to changes in network conditions, requiring adaptive mechanisms for optimal operation. Overhead: Additional signaling, control information exchange, and processing requirements may impact energy consumption and network efficiency. Fairness: The protocol may not provide strict fairness in channel access opportunities for all devices, especially in congested networks. Limited Analytical Insights: The paper primarily focuses on protocol design and simulation-based performance evaluation, lacking in-depth analytical insights into theoretical properties and performance bounds. Despite these limitations, the distributed queueing-based random access protocol presents a promising approach to address challenges in LoRa networks, offering enhanced efficiency and scalability for IoT applications. In , Abedi and Pourhasani present a paper titled "Prioritized Multi-Channel MAC Protocol in Ad Hoc Networks Using a TDMA/CSMA Approach," aiming to address the demand for efficient medium access control (MAC) protocols in ad hoc networks. Emphasizing the importance of channel prioritization and hybrid TDMA/CSMA techniques, the paper proposes a prioritized multi-channel MAC protocol. This protocol integrates Time Division Multiple 26 | P a g e Access (TDMA) and Carrier Sense Multiple Access (CSMA) methods to enhance network performance by improving channel utilization and reducing contention, thereby boosting overall throughput and reliability. The proposed MAC protocol offers benefits such as H increased network capacity, reduced packet collisions, and enhanced Quality of Service (QoS). A By prioritizing channels and employing a hybrid TDMA/CSMA approach, the protocol aims P to optimize resource utilization, particularly in scenarios with dynamic traffic patterns. However, challenges such as increased implementation complexity and variability T in E research effectiveness based on network topology and traffic characteristics may arise. Further and experimentation are necessary to comprehensively evaluate the performance R of the prioritized multi-channel MAC protocol. 1 In paper , Distributed Queuing (DQ) emerges as a promising solution, offering collision- free contention-based random access to enhance the scalability of LoRa networks compared to the default Aloha protocol. The fundamental idea behind DQ involves organizing nodes into a virtual queue using a collision resolution algorithm and servicing them sequentially to access the channel, thus mitigating collisions typical of Aloha. DQ presents several advantages, including near-optimal throughput performance irrespective of traffic load, bounded access delay, and decentralized operation without necessitating synchronization. However, existing DQ schemes, such as the Collision Tree Algorithm, were initially devised for cable TV networks featuring full-duplex links, rendering them inefficient for the half-duplex LoRa environment due to high signaling overhead stemming from the contention resolution process. Extending DQ to multichannel LoRa networks poses another challenge, namely achieving balanced load distribution across channels. Despite DQ's superiority over Aloha in dense LoRa deployments, meticulous protocol design is imperative to realize these advantages, particularly considering LoRa's half-duplex nature and utilization of multiple channels. The paper introduces a novel Medium Access Control (MAC) protocol named Hybrid ALOHA (H-ALOHA), which amalgamates the Pure ALOHA (P-ALOHA) and Slotted ALOHA (S-ALOHA) protocols to cater to specific needs in wireless networks, including energy conservation, delay reduction, and throughput enhancement. The fundamental idea is to capitalize on the respective strengths of both protocols: P-ALOHA exhibits superior energy efficiency and packet delivery for individual transmissions, while S-ALOHA generally outperforms P-ALOHA. To evaluate the steady-state performance of H-ALOHA comprehensively, the paper develops a finite-state Markovian model, encompassing metrics such as normalized throughput, backlogged throughput, access delay, backlogged delay, and energy consumption. Through simulations, the proposed hybrid protocol demonstrates 27 | P a g e significant advantages, surpassing S-ALOHA by 60% in normalized throughput, 15% in access delay, and 23% in total energy consumption during transmission on average. However, it's important to note a potential limitation: the analysis is confined to specific scenarios and H assumptions, and the protocol's performance might vary under different wireless network A conditions and requirements. P T domains, In , the Internet of Things (IoT) has emerged as a pivotal technology across various ranging from smart cities to industrial automation. Efficient Medium Access Control E (MAC) protocols play a critical role in ensuring reliable and energy-efficient data transmission within R IoT networks. One promising approach to tackle the challenges inherent in IoT environments 1 is the utilization of Distributed Queuing (DQ) MAC protocols, which aim to mitigate collisions and enhance system throughput. The paper discusses the Q-learning-based Distributed Queuing Medium Access Control (QL-based DQMAC) protocol tailored for IoT networks, which endeavors to optimize the number of contending nodes in each contention period to reduce collision probability. The protocol leverages the Q-learning algorithm, empowering each node to compute its active rate autonomously and decide whether to remain active or enter a sleep mode in the subsequent contention period. This adaptive behavior contributes to improvements in throughput, delay, and energy consumption. However, the protocol's limitations include its lack of consideration for traffic prioritization, potentially hindering its ability to effectively manage the diverse Quality of Service (QoS) requirements characteristic of IoT applications. In , H. Hassen, S. Meherzi, and Z. Ben Jemaa delve into the realm of Software-Defined Networking (SDN) to tackle the pressing issue of optimizing multipath routing. Recognizing the paramount importance of efficient resource allocation and heightened network performance, the paper introduces refined strategies for multipath routing based on Q-learning. These strategies aim to enhance exploration mechanisms to achieve more effective resource utilization and bolster overall network efficiency within SDN environments. The proposed approach promises several benefits, including heightened network efficiency, congestion reduction, and improved quality of service. By employing innovative exploration strategies, the study seeks to alleviate network bottlenecks and elevate the overall user experience within SDN networks. However, the paper acknowledges certain limitations, notably potential scalability issues and overhead associated with implementing Q-learning in large-scale SDN networks, underscoring the need for further exploration and optimization in this domain. 28 | P a g e  Approach Advantages Disadvantages EQL-DQMAC (proposed) Adaptive optimization, energy- Complexity, overhead, tuning H efficient, Reduced contention challenges, scalability and delay concerns A QL-based DQMAC Optimized contending nodes, No P QoS prioritization, throughput, delay, energy limitations Multichannel Aloha Increased throughput, T High collision rates, Lack of Improved load balancing, sensing mechanisms,E Inability Reduced delays to adapt SARSA(λ) Convergence, adaptability, Complexity, R exploration- resource allocation exploitation 1 trade-off, scalability, lacks guarantees RC-SFAMA Throughput in dense networks, Complexity, synchronization, energy-efficient network sensitivity Energy-efficient MAC Energy efficiency, lifetime Dynamic IoT limitations, QoS protocols limitations Distributed Queueing Channel access, scalability, Complexity, for LoRa Networks latency, energy synchronization, dynamics sensitivity, overhead, fairness Alternative MAC Throughput, spectral efficiency, Protocols QoS, adaptability Complexity, overhead Adaptability, QoS differentiation, resource Computational overhead, RL-MAC utilization reward design, single-hop Distributed Queue- Comprehensive framework, Parameter optimization, LTE based Random Access Effective contention resolution, access impact, Traffic pattern Framework Dynamic