Page 1 of 1
How to write binary data to a file?
Posted: Tue Jul 26, 2016 12:17 pm
by David241
All I can find in the PHP Manual for writing raw bytes (unsigned integers) to a file is fwrite. But when I use it, it seems to write characters, not raw bytes. For example, the value 1 (0b1) is actually written as 0x31, not 0x01 as desired. Yes, the fopen was done with the "b" mode specified.
It doesn't work this way in C or C++.
Web searches turn up nothing useful on this subject.
I have posted this question on two other fora, and have received unhelpful answers, so please read the question carefully.
System: Windows 8.1, PHP 5.5.10
Re: How to write binary data to a file?
Posted: Tue Jul 26, 2016 7:16 pm
by Christopher
I am guessing that the unhelpful answers told you that there is no such thing as a binary file -- only binary data. And that is it applications that interpret that data as being in a specific format.
That said, from your post it seems that the file may be being marked as data (That's a Windows thing, or used to be. Unix does not care.), but that the data inside the file is not in the binary format you expect. I think the first thing I would look into are the pack() and unpack() functions. They will create a binary string that can be written with fwrite().
If you post a little more information about the data and specifically what you are trying to do, I'm sure we can get it working. If you have some test code, I'd be glad to try it on Windows and Linux.
[Solved] Re: How to write binary data to a file?
Posted: Wed Jul 27, 2016 6:07 am
by David241
I have received good help on this topic from the PHP News Group (mailing list). Although the PHP Manual appears to document fwrite as writing either character strings or binary data, in reality it will write only character strings. For example, if you write the integer or binary value 0x1 as a byte, the value 0x31 will actually be written, regardless of whether the 'b' mode is specified in fopen.
One solution is to use fprintf. For example,
will actually write the byte value 0x1 at the current offset in the file.
Re: How to write binary data to a file?
Posted: Wed Jul 27, 2016 6:22 pm
by Christopher
So I wrote a test program to see if I could read and write binary data to/from a file. It seems to work fine. The pack() function is interesting because you could use it to convert integers and floats to binary data with specific byte orders. So there is a lot of control of how the binary data is created.
Here is the test program I wrote. I'd be interested to see if it works for you.
Code: Select all
<?php
$data = pack("c*", 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0a);
$filename = 'data.txt';
$fp = fopen($filename, 'wb');
fwrite($fp, $data);
fclose($fp);
$size = filesize($filename);
$fp = fopen($filename, 'rb');
$data = fread($fp, $size);
fclose($fp);
$n = strlen($data);
for ($i=0; $i<$n; ++$i) {
echo ord(substr($data, $i, 1)) . ' ';
}
The output, as expected, was:
1 2 3 4 5 6 7 8 9 10
Re: How to write binary data to a file?
Posted: Wed Jul 27, 2016 6:36 pm
by David241
Christopher, Congratulations on apparently getting fwrite to write binary bytes instead of forcing it to a character string. Perhaps this implies that the fault in my original posting wasn't fwrite itself, but an implicit type conversion? Not sure, but since I now have a program that is short (elegant) and works (I've tested extensively), I'm happy and don't need to investigate pack().
One would think that a Swiss army knife language like PHP would have a good reliable and documented way to read and write integers, but I guess they had to forget something. Actually, I've found lots of other problems in PHP that I had to work around (including one that discarded important business data for two years when they changed how a basic function worked without any notice), yet I'm still mostly happy with PHP.
Re: How to write binary data to a file?
Posted: Wed Jul 27, 2016 8:01 pm
by Christopher
David241 wrote:Christopher, Congratulations on apparently getting fwrite to write binary bytes instead of forcing it to a character string. Perhaps this implies that the fault in my original posting wasn't fwrite itself, but an implicit type conversion? Not sure, but since I now have a program that is short (elegant) and works (I've tested extensively), I'm happy and don't need to investigate pack().
Yeah, a little code is worth a thousand words. Glad I could help.
I know I have used pack()/unpack() in the past, but apparently is wasn't worth remembering. The manual page for those functions is pretty clear. If you need to write multi-byte numbers there is support of doing that. I recall that I had to control byte order for data for another system and it was pretty easy to do.
David241 wrote:One would think that a Swiss army knife language like PHP would have a good reliable and documented way to read and write integers, but I guess they had to forget something. Actually, I've found lots of other problems in PHP that I had to work around (including one that discarded important business data for two years when they changed how a basic function worked without any notice), yet I'm still mostly happy with PHP.
Well, I would not call PHP a Swiss army knife language. That sounds more like C/C++ or Python. I think 99% of the PHP in the world is just generating HTML to send out on port 80. And your experience confirms that there are very few PHP programmers who could give you advice on writing binary data. That's just not in the set of problems PHP is designed to solve -- which are all very text oriented. And the fact that PHP has the C functions as the way to build binary data is pretty typical.
Re: How to write binary data to a file?
Posted: Tue Aug 02, 2016 6:43 pm
by Vegan
Binary files are a dinosaur, use XML like everyone else does, this way when the next developer gets to it, it's easier to figure out
PHP is simply server side web host, it was not ever designed as a general purpose interpreter
if want to do general development, there are lots of other programming languages available
PHP and JavaScript were deigned with syntax similar to C and C++ for reason, Linux and Windows are both written largely with C++
Re: How to write binary data to a file?
Posted: Tue Aug 02, 2016 10:33 pm
by David241
Vegan wrote:Binary files are a dinosaur, use XML like everyone else does
Binary files have the advantage of efficiency and are useful when human readability is not needed.
I don't think it is true that everyone uses XML. Many recent projects use lots of data interchange languages, and I suspect that JSON is a bit more popular than XML due to its superior readability, compactness, and generality. XML uses a verbose syntax, and in its full form requires a data definition.
this way when the next developer gets to it, it's easier to figure out
Self-defining data is useful in some applications. But, in most applications, it is sufficient to describe the data formats used in a manual or other accompanying documentation, for those who know how to write clearly.
PHP is simply server side web host, it was not ever designed as a general purpose interpreter
I don't know the history of PHP; you may be right. But however it was designed, its practical use is as a server-side programming language, called by the server software. It is probably used almost as often as Javascript and other client-side languages to solve real-life problems. It has the advantage of better security and wider functionality than Javascript.
if want to do general development, there are lots of other programming languages available
Quite true. Also irrelevant.
Why did I post this thread? Because I run a company that intends to have about a million customers in its next growth spurt. Each client is given a random client ID, which is a key into the database. For security the database itself is stored off the Internet. But for the sake of allocating new client IDs, the server needs to know all the existing client IDs. How to solve this problem? The most efficient solution in space and time is to use a long bit array to represent the set of existing client IDs. A simple SQL statement builds the set array from the offline database. Then an online PHP program allocates a random client ID, checking with the online set to ensure uniqueness.
Solving this problem using XML to represent the set of up to a million client IDs might require a million XML elements. In the efficient binary solution is requires only a 122 KB file, which is a very modest size.
I'm a retired software engineer. The advantage of professional training in computer science is that you have the ability to choose the right programming language, data structures, and program functions for the problem to be solved. Knowledge gives you the ability to rise above the limitation of working within a narrow set of 'favorites' and instead using the tools that are best suited to the problem domain.
C and C++ don't handle XML as part of the language, but they do handle both binary and string I/O using files. PHP can do the same thing, but the documentation is not too helpful. These postings of mine provide one approach (another is to use fwrite with the CHR function applied to a binary argument).
Thus, this thread is a key for those programmers using PHP who need or desire the efficiencies that only binary files can provide.
Re: How to write binary data to a file?
Posted: Wed Aug 03, 2016 12:52 pm
by Christopher
I was sort of with you up until this point:
David241 wrote:Why did I post this thread? Because I run a company that intends to have about a million customers in its next growth spurt. Each client is given a random client ID, which is a key into the database. For security the database itself is stored off the Internet. But for the sake of allocating new client IDs, the server needs to know all the existing client IDs. How to solve this problem? The most efficient solution in space and time is to use a long bit array to represent the set of existing client IDs. A simple SQL statement builds the set array from the offline database. Then an online PHP program allocates a random client ID, checking with the online set to ensure uniqueness.
That is a solution, but it seems a little odd and over engineered. Typically now these kinds of unique IDs would be generated with a hash function. It seems like these systems could need just share is the current auto increment key and unique IDs would be available to both systems. A secure web service might be a better approach.
Re: How to write binary data to a file?
Posted: Wed Aug 03, 2016 6:41 pm
by David241
[quote Christopher]unique IDs would be generated with a hash function[/quote]
A great idea, which I have sometimes used. In this case, 2000 clients already exist, each with their own randomly-generated client ID.
I was only presenting the problem domain to motivate my use of binary set representations, which had been attacked.
In general, I will argue that there will always be a place for reading and writing binary data using files.
Thanks,
David
Re: How to write binary data to a file?
Posted: Thu Aug 04, 2016 2:13 pm
by Christopher
David241 wrote:A great idea, which I have sometimes used. In this case, 2000 clients already exist, each with their own randomly-generated client ID.
If I was going from 2K to 1M users, I'd just upgrade those users to the new IDs. But, I prefer to do the work up front to get everything in the best current "right way."
David241 wrote:I was only presenting the problem domain to motivate my use of binary set representations, which had been attacked.
In general, I will argue that there will always be a place for reading and writing binary data using files.
Of course -- hence the pack()/unpack() functions. I think the point was that almost always with PHP, the program is passing text to another application (e.g., database) that may internally store binary data. I think I've only generated or read binary data once or twice in PHP. And your problem would be solved by most PHP programmers in a way that did not involve binary data. I don't know enough about your problem to judge the design one way or the other. I'm only glad we could help you sort out this issue.