Master synchronization and timing to handle dynamic web content
Web scraping and automation require proper timing to handle dynamically loaded content. Zendriver provides multiple strategies for waiting and synchronization.
Many Zendriver methods automatically wait for elements to appear:
import asyncioimport zendriver as zdasync def main(): browser = await zd.start() tab = await browser.get("https://example.com") # These methods wait up to 10 seconds by default button = await tab.find("Submit") # Waits for text input_field = await tab.select("#email") # Waits for selector all_links = await tab.select_all("a") # Waits for at least one matchif __name__ == "__main__": asyncio.run(main())
Adjust the default timeout for specific operations:
# Wait up to 30 seconds for a slow-loading elementbutton = await tab.find("Load More", timeout=30)# Short timeout for optional elementstry: ad = await tab.select(".advertisement", timeout=2) await ad.remove_from_dom()except asyncio.TimeoutError: print("No ad found, continuing...")
# Wait 2 secondsawait tab.sleep(2)# Short pause for animationsawait tab.sleep(0.5)# Wait and update referencesawait tab # Calls tab.wait(), which includes a small sleep
Calling await tab updates internal references and allows the browser to “breathe”. Use it when experiencing timing issues.
# Wait until page is interactiveawait tab.wait_for_ready_state("interactive", timeout=10)# Wait until page is fully loadedawait tab.wait_for_ready_state("complete", timeout=15)# Wait until page starts loadingawait tab.wait_for_ready_state("loading", timeout=5)
Available states:
"loading" - Document is still loading
"interactive" - Document is parsed but sub-resources are loading
"complete" - Document and all sub-resources are loaded
The get() method automatically waits for navigation and DOM events:
# Navigate and wait for page loadawait tab.get("https://example.com")# The page is ready after get() completessearch_box = await tab.select("input[type='search']")
# Click button that loads contentload_button = await tab.find("Load Products")await load_button.click()# Wait for the content to appearfirst_product = await tab.select(".product-card", timeout=10)# Now safe to query all productsproducts = await tab.select_all(".product-card")
# Wait for status to change from "Loading..."while True: status = await tab.select(".status") if status.text != "Loading...": break await tab.sleep(0.5)print(f"Final status: {status.text}")