8

I have an arducam mini 2MP camera connected to an ESP8266 (12-E) module and I am trying to implement video streaming inside a window with some text and control buttons around it, all in the same browser tab /page. I have created two HTML pages for the server to use. The first is the home webpage with no image streaming, just a simple page with text buttons and some CSS. The second HTML page serves the continuous frames (streaming video) along with some text and buttons to the browser. When the home page is sent to the browser everything is displayed the way I expect. But, when the second HTML webpage is served but some strange things happen when the browser(Firefox or Chrome) receives the reply from the server (esp12-e).

Normally I would expect a small window displaying continuous frames taken from the camera with some text over that window an some control buttons under it. But, instead of that two things happen.

  1. Only the video streaming window is displayed in the browser's tab but around this window there is only a gray background color. No buttons no text. When I open the HTML Inspector, inside "head", there are a few lines of HTML code that create the background gray color and some CSS stuff I have not written in my server. Somehow the browser creates this lines of code automatically and adds them in my original HTML code.
  2. In my original HTML code, inside "body", along with the code for the streaming window, I have the code for the text and button elements that will be displayed. But in the browser, these parts disappear. When I open the Inspector these elements don't exist! I have tried various approaches so far to avoid this situation by isolating/embedding the streaming window inside the browser's tab. These approaches are: iframe, data URI, multipart/x-mixed replace, form. Unfortunately the same result occured for all of these approaches ( gray background color, streaming window in the middle of the screen and disappeared buttons and text ).

The only thing I know is that, when the browser "sees" the incoming image from the server it produces these side effects. When I create an HTML only with text and buttons it is displayed just fine. I do something the wrong here but I cannot find what it is.

Below I attach 2 pictures of what I get in the browser's tab and the HTML code I send from the esp-12e server for a photo capture

void serveWebpage(WiFiClient client){

  String answer = "HTTP/1.1 200 OK\r\n";     
  answer += "Content-Type: text/html\r\n\r\n";
  answer +="<!DOCTYPE HTML>\r\n"; 

  answer += "<html>\r\n";
  answer +="<head><title> Monitor </title></head>\r\n";

  answer += "<body>\r\n";
  answer += "<h1 style=\"position:relative; left:25px;\"> &#9875     Observation Panel &#9875</h1>\r\n";     // Header Text

  answer += "<a href=\"/videoStream\"><button type=\"button\"  style=\"position:absolute; top:340px;";   // First Button
  answer += "left:95px; color:blue; height:70px; width:90px; font-weight: bold; border-style:outset;";
  answer += "border-width:2px; border-color:black;\"> Video Stream </button></a>\r\n";

  answer += "<a href=\"PhotoCapture\"><button type=\"button\"  style=\"position:absolute; top:340px;";   // Second Button
  answer += "left:195px; color:blue; height:70px; width:90px; font-weight: bold; border-style:outset;";
  answer += "border-width:2px; border-color:black;\"> Video Stream </button></a>\r\n";

  answer += "<div>\r\n";
  answer += "<img src='data:image/jpeg; charset=utf-8; base64,";   // Here the image is wrapped with data URI to display it in the browser


       myCAM.clear_fifo_flag();    // this part is taken from the arducam library exammples. It captures the image and sends it to browser
       myCAM.start_capture();       

       while (!myCAM.get_bit(ARDUCHIP_TRIG, CAP_DONE_MASK));  // wait here until capture has completed
       size_t len = myCAM.read_fifo_length();

       myCAM.CS_LOW();         
       myCAM.set_fifo_burst();     

       #if !(defined (ARDUCAM_SHIELD_V2) && defined (OV2640_CAM))
       SPI.transfer(0xFF);
       #endif   

       static const size_t bufferSize = 4096; //4096
       static uint8_t buffer[bufferSize] = {0xFF};

       while (len) {                
         size_t will_copy = (len < bufferSize) ? len : bufferSize;
         SPI.transferBytes(&buffer[0], &buffer[0], will_copy);
         if (!client.connected()) break;

         client.write(&buffer[0], will_copy);
         len -= will_copy;
       }
  myCAM.CS_HIGH();

  answer +="9k=' />"; // closing the <img> 
  answer +="</div>\r\n";
  answer +="</body>\r\n";
  answer +="</html>\r\n\r\n";

  client.print(answer);  
}  

Video Window_missing buttons and text HTML code that I have written to be served by server esp-12e

I made some progress finally but not 100%. I managed to display jpeg images in the iframe by embedding data in jpeg format from an image with data URI method inside Iframe element.

    string = "<iframe srcdoc='<img src=\"data:html/text;base64,/9j/4AAQ..... \" > ' > "; 

My mistake was that I did not use the quotes with the correct order and image data were interpreted as text in the browser. Then I tried to do the same thing with the function I use to send the captured image from the camera to the browser. Unfortunately the same issue arose and I cannot fix it this time. Something happens when I send a string with multiple quotes to the browser because it interprets them as text and not as jpeg data format like this: /9j/4AAQ...... I uploaded a picture from my browser's inspector(showing the browser's received data when I use the function for camera's sent frame data) to figure out easier what I mean. Any ideas on this ?

Data are interpreted faulty because of multiple quotes(?)

Here is a review of what I have completed so far. I created an HTML with an Iframe inside it and also some buttons. Both iframe and buttons are displayed correctly in the same browser's tab. Now, Inside iframe I put srcdoc attribute and inserted the raw jpeg data directly in there (of a sample jpeg image) since they are encoded (base64) but the browser interpreted these jpeg data as plain text and displayed them in the iframe as text. Then I used the image tag inside srcdoc to wrap the raw jpeg data in the iframe. This worked after some mistakes I did with the quotes inside iframe string. Then I removed the raw jpeg data from the image tag and replaced them with the function that brings jpeg data from the camera. I send the first part of the answer string (opening iframe and img tags), then I send the data from the camera and finally I send the second part of the answer string (closing iframe and img tags). Normally it should work since I followed the same procedure as before. But the browser could not interpret the image ...again.

Below, I have added the code parts for the encoded sample image (that the browser interpreted as image) and then the camera function (which interpreted them as odd characters and not image), for comparison. Both should work the same way but only the first works.

Encoded sample image:

    answer = "<iframe srcdoc='<img src=\"data:image/jpeg;base64,/9j/4AAQS...0KDQo=\"> ' scrolling=\"no\" width=\"340\" height=\"340\" >  <p> Error </p> </iframe>\r\n ";

Camera function sendFrame():

    answer = "<iframe srcdoc=\"<img src='data:image/jpeg;base64,";
    client.print(answer);
    sendFrame();
    answer ="' > \" >  <p> Error </p> </iframe>\r\n ";
    client.print(answer);

So, I think I have found what goes wrong with the camera's incoming jpeg data The camera function brings the jpeg data (to the server and then to the client) in a format that the browser interprets it as text or something like that because it contains strange characters (check the last image I posted).

Also in order to write the html code I use quotes " and ' (or " and \') to create the iframe code and everything else inside iframe.

And here is the thing: Because some of the camera's jpeg data are interpreted as quotes by the browser, they interact with the quotes I put inside iframe to wrap the img tag and the data coming from the camera and That is why it messes everything up in iframe (I think)

Is there anyway to convert the image data coming from the camera function into base64, so as they don't interact with the wrapping quotes of the iframe and image tags?

  • 2
    Is it possible for you to include a minimal code example of the issue here? I suspect that without that, any answer would just be speculation. My guess is that maybe you're only serving the image, or serving it with the wrong Content-Type, as it's clearly being interpreted as an image (all the inserted styles are internal Firefox styles) – Aurora0001 Jul 16 '17 at 19:21
  • The second image includes the code but I will upload one more. Thanks for the help – Sparky Jul 16 '17 at 19:33
  • 1
    I was thinking of the server code too, if you have it. Are you using a premade library like this or did you roll your own? – Aurora0001 Jul 16 '17 at 19:38
  • 1
    Yes I read that code but I thought to build it by my own ( so as to understand it better). I will upload it in a moment – Sparky Jul 16 '17 at 19:47
  • 1
    Excuse me for the delayed answer but I was tidying up the code a bit so as to be... readable. I tried different methods as I mentioned before. But the same result occured every time. So it must be something else . I am a little lost here – Sparky Jul 16 '17 at 20:25
4

You are writing directly from your web camera the data to the stream and after that comes the empty answer template you are constructing.

You can see, you do not write the first part of answer to wifi and you write data from camera directly to the wifi.

Because your data from camera has already a formatting, I would suggest adding data to iframe instead of img tag, where it is now and does not belong to.

void serveWebpage(WiFiClient client){

  String answer = "HTTP/1.1 200 OK\r\n";     
  answer += "Content-Type: text/html\r\n\r\n";
  answer +="<!DOCTYPE HTML>\r\n"; 

  answer += "<html>\r\n";
  answer +="<head><title> Monitor </title></head>\r\n";

  answer += "<body>\r\n";
  answer += "<h1 style=\"position:relative; left:25px;\"> &#9875     Observation Panel &#9875</h1>\r\n";     // Header Text

  answer += "<a href=\"/videoStream\"><button type=\"button\"  style=\"position:absolute; top:340px;";   // First Button
  answer += "left:95px; color:blue; height:70px; width:90px; font-weight: bold; border-style:outset;";
  answer += "border-width:2px; border-color:black;\"> Video Stream </button></a>\r\n";

  answer += "<a href=\"PhotoCapture\"><button type=\"button\"  style=\"position:absolute; top:340px;";   // Second Button
  answer += "left:195px; color:blue; height:70px; width:90px; font-weight: bold; border-style:outset;";
  answer += "border-width:2px; border-color:black;\"> Video Stream </button></a>\r\n";

  answer += "<div>\r\n";
  answer += "<iframe src="/yourURIforImage"></iframe>\r\n"; 
  answer +="</div>\r\n";
  answer +="</body>\r\n";
  answer +="</html>\r\n\r\n";

  client.print(answer);  
}  

void serveImage(WifiClient client) {
       myCAM.clear_fifo_flag();    // this part is taken from the arducam library exammples. It captures the image and sends it to browser
       myCAM.start_capture();       

       while (!myCAM.get_bit(ARDUCHIP_TRIG, CAP_DONE_MASK));  // wait here until capture has completed
       size_t len = myCAM.read_fifo_length();

       myCAM.CS_LOW();         
       myCAM.set_fifo_burst();     

       #if !(defined (ARDUCAM_SHIELD_V2) && defined (OV2640_CAM))
       SPI.transfer(0xFF);
       #endif   

       static const size_t bufferSize = 4096; //4096
       static uint8_t buffer[bufferSize] = {0xFF};

       while (len) {                
         size_t will_copy = (len < bufferSize) ? len : bufferSize;
         SPI.transferBytes(&buffer[0], &buffer[0], will_copy);
         if (!client.connected()) break;

         client.write(&buffer[0], will_copy);
         len -= will_copy;
       }
       myCAM.CS_HIGH();
}

and where you verify to which uri you responded with serveWebpage() you make similar to match the serveImage() and /yourURIforImage.

  • I followed your opinion and I added one client.print(answer) immediately after the first part of answer and now I can see some progress. I kept the img element to see what is going to happen. The text and button are displayed correctly (that is a good sign I think) but the image data are not interpreted by the browser as an image , but odd characters. Why the browser does not understand the image data as image since I converted the data to base64? I will add an iframe and erase the img element to see what will happen and I will be back – Sparky Jul 21 '17 at 23:17
  • I created an iframe window and in there i tried to embed the image taken from the camera with data URI method. But the image data are not interpreted as image by the browser and strange characters are displayed inside iframe. Also I used srcdoc and not src to embed the image data. Any ideas? – Sparky Jul 23 '17 at 18:44
  • To make this work, write first a method that successfully returns the image only and then in another method call that method < iframe src=" path to the image > < / iframe > – mico Jul 26 '17 at 6:35
  • Yes it is not bad idea but the thing is that I used an image from the camera the following way: I saved the captured jpeg picture in my pc, then I encoded it to base64 and finally I put it in the iframe srcdoc data URI to be displayed. – Sparky Jul 26 '17 at 8:58
  • I also put the image data directly inside srcdoc without image tag but the browser interprets them as characters. Odd characters not just letters or numbers. – Sparky Jul 26 '17 at 9:11
3

The thing that stands out is your statement:

Something happens when I send a string with multiple quotes to the browser because it interprets them as text and not as jpeg data format like this: /9j/4AAQ......

I remembered there is a specific way to embed quotes is html: some sort of escape characters are needed. I googled

embed quotes in html

This question was answered on StackOverflow here:

How to properly escape quotes inside html attributes?

It looks like you have quotes in quotes in quotes. So I am not sure if you are employing the escape characters already or not, but I would suggest using the explicit option

&quot; 

for your embedded quotes.

  • I will try this and I tell you what happened. &quote is the same as \" or '? If I use &quote and \" and ' in the same string how the browser is going to interpret them. Thanks – Sparky Jul 26 '17 at 8:48
  • Not sure. But there were many results returned from a Google search. So I would bet the answer to that question to be found quickly, if you do the search. – grldsndrs Jul 26 '17 at 14:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.